Context
Audio description narrates what’s happening on screen for people who can’t see it. WCAG requires it for prerecorded video at level AA (success criterion 1.2.5).
At Norton, audio description was the most labour-intensive accessibility remediation task there was. It meant specialised vendor work that was expensive and slow, and because of the cost, many videos never got it. I worked with teams so that new videos would get human-created audio description, which is the ideal. That still left a backlist of hundreds of existing videos with none.
I designed the workflow and built the prototype.
The challenge
- Cost and scale. Vendor description wasn’t affordable across hundreds of videos.
- Old videos weren’t made for it. Many backlist videos were made years ago, with few quiet moments where a description could fit.
- Generic output. My biggest worry was that AI descriptions would be generic filler, or would describe details and miss the point of the scene.
- Copyright and processing cost. The videos are copyright-protected content, and processing video through a hosted model is expensive.
What I did
- Worked with teams to get human-created audio description on new videos, and scoped the prototype to the backlist.
- Built a first version with cloud text-to-speech, and an algorithm to fit descriptions into the audio.
- Rebuilt the description step to work in passes, so descriptions follow the story of the video, not just the frame.
- Moved playback to an accessible player that pauses the video while each description plays.
- Ran the model locally, and required a person to review every description before publishing.
- Piloted it on 200 videos, and reviewed every description myself.
How the program ran
I scoped the work to the backlist, since new videos would get human-created description instead. I built and tested two versions, then piloted the second on 200 videos. I reviewed every description myself, and measured the pilot by how many descriptions needed revision. After the pilot, the prototype was shelved for now because of legal and liability concerns.
First version: text-to-speech and audio ducking
The first prototype voiced descriptions with Amazon Polly, a cloud text-to-speech service, and that worked for many videos. The problem was older videos that had no quiet stretches to put a description in. I explored audio ducking, lowering the video’s soundtrack under the narration, and wrote an algorithm that analysed the captions to find the best insertion points. It worked, but not consistently.
The descriptions themselves had a bigger problem. They described a single still image, so they missed any action, and they missed the bigger picture of what the video was about.
Second version: descriptions that follow the story
I rebuilt the description step to work in passes, with a person reviewing the result before anything is published. The dashed steps under Publish are a change I’d make next, for videos with no sound (see the end of this case study).
Text description: How a description gets from video to viewer
The flow has three phases: draft, review and publish.
-
Draft (automated, on a local model), from a backlist video
and its captions:
- Read the whole transcript and write a short summary of the video.
- Find key moments: scene changes that fall in gaps between captions.
- Describe each moment from three frames, using the summary and nearby dialogue.
- Remove repeats: the model can skip a moment, and a check in code drops repeats.
- Draft the description file, plus a report for the reviewer that lines up captions and descriptions in time order.
- Review (a person): every description is reviewed and edited if needed. Nothing is published until it's approved.
-
Publish, depending on the audio. Routing by audio is
planned, not built.
- Videos with sound are published in AblePlayer, which pauses the video and reads each description aloud. This is how the prototype handles every video today.
- Videos with no audio would get a generated audio clip of the narration, timed to the video, so they aren't lengthened by pausing. The clip would be added as a parallel audio track that viewers can turn on or off. (Planned, not built.)
- Read the whole transcript first. Before looking at any frame, the model reads the full caption file and writes a two- or three-sentence summary of what the video is about. Every description after that uses it as context.
- Find the key moments. Scene detection flags cuts and big visual changes. A moment becomes a description only if it falls in a gap between captions, so descriptions never talk over speech.
- Describe a sequence, not a still. For each gap, the model sees three frames, from the start, middle and end, so it can describe action and change, not just a single image.
- Keep the narrative in view. Each request also carries the video summary, the dialogue just before and just after the gap, and the model’s recent descriptions, so each new description fits the story so far.
- Remove repeats. The model skips a gap when the dialogue already covers it or nothing new is visible, and a separate check in code drops descriptions that repeat the previous one or the captions.
The prompt holds the model to the conventions of professional description: present tense, only what’s visible, no mood or guessed intent, no race or ethnicity, on-screen text read as it appears, and charts described by their form before their trend. Those are instructions to a model, not guarantees, which is why every description is reviewed.
Pausing instead of ducking
For playback I moved to AblePlayer, an accessible open-source media player. It reads each description aloud and pauses the video while it plays. That gave a clean, clear narration with no risk of talking over important content, and it worked more consistently than ducking. Descriptions also stay as editable text, so a reviewer can fix one in a text file.
Running it locally
The model ran locally, using a Gemma model from Google. That kept the cost of processing video down while it was a prototype, and it meant no copyright-protected content was uploaded to a hosted AI service.
A person reviews every description
Output was always a draft. Alongside the description file, the tool produced a report that lines the captions and descriptions up in time order, so a reviewer could read them together. Nothing was published without human review.
Impact
- The pilot worked. Across 200 videos, 86%+ of the drafted descriptions needed no revision.
- It was shelved for now, because of legal and liability concerns.
- It showed a workable path for video that can’t be significantly changed after the fact.
- Not measured: reviewer time, or cost compared with vendor description.
- Not independent: I built the tool and reviewed its output myself. An independent reviewer, such as a professional describer or a blind viewer, would be the next step before relying on the 86% figure.
What I’d do differently
Design audio description into content from the start. Audio description works best when it’s planned as the video is made, not added afterwards. Older videos with no quiet moments showed the limit of fitting it in later. The prototype is still a sound approach for most of the backlist, where the content can’t be significantly changed, but for new video the answer is to plan for description from day one.
Match playback to the video. Some backlist videos had no audio at all. Pausing those to read each description makes them longer than they need to be, because there’s no sound to talk over. Next time I’d route each video by whether it has sound. Silent videos would get a generated audio clip of the narration, timed to the video and added as a parallel audio stream that viewers can turn on or off. Videos with sound would keep AblePlayer’s pause-and-read approach.