Narrated Shorts: Word-Synced Subtitles

Why this exists

I posted 48 short videos before noticing every one of them was silent. They were card news images stitched into a slideshow. I knew there was no audio and figured music would cover it later. Adding narration changed the format entirely — the same content became easier to follow, because you can listen instead of reading every frame.

The hard part was never the voice. It was keeping subtitles in sync with it. Time them by hand and they hold until you change the voice, adjust the pace, or rewrite one line. Then every timing is wrong again, and after the third round you stop making videos.

How it works

This pipeline never times anything by hand. It generates the voice, then listens back to it with speech recognition to recover when each sentence was spoken. Timing is a derived value, so it survives any change to voice, speed, or wording.

script.txt --> TTS --> voice.mp3 --> speech recognition --> timing.json
                                                                |
lines.json ------------------ align <---------------------------+
                                |
                           props.json --> Remotion --> short.mp4

You write two short files. Everything else runs from one command.

node scripts/make-short.mjs <job-folder>

Output: 1080x1920 MP4, roughly 30 seconds for an 11-line script.

The two files

script.txt is what the voice says. lines.json is what the screen shows.

They are separate on purpose. TTS mispronounces 2,645 — it needs the number spelled out. But the screen wants 2,645, because nobody scans spelled-out numbers. One text cannot serve both.

[
  { "text": "2,645 hits\nin 142 files", "hi": ["2,645"] }
]

hi marks words that get an accent chip wiping in left to right as they are spoken.

What is included

• 4 scripts — voice generation, speech recognition, subtitle alignment, and the runner that chains them
• 3 Remotion compositions — the video itself, four background styles, and the project root
• doctor.mjs — checks Node, ffmpeg, Python, the recognition package, your API key, and the Remotion project, then prints the exact fix for anything missing, matched to your OS
• setup-remotion.mjs — creates the Remotion project and wires the compositions in
• 4 reference documents — script structure, subtitle motion, environment setup, and the failures this pipeline already hit

Decisions that came from failures

Pace is a target rate, not a fixed multiplier. The same script came back at 25.8s, 31.4s, and 39.7s across three runs. A hardcoded speed multiplier therefore produces a different pace every time. The pipeline targets a characters-per-second rate and solves for the multiplier from the actual audio.

Subtitles over artwork need a stroke, not a scrim. Half your images will be light, and white text vanishes on those. A dark band behind the text fixes contrast but covers the artwork you are showing. A black stroke drawn behind the glyphs keeps both.

Desktop screenshots are unreadable in vertical video. A browser-width capture scaled into a 1080-wide frame renders body text at a few pixels. Cropping vertically does not help, because the scale is set by width.

Linear motion reads as machine-made. Every transform eases out. Words enter staggered rather than as a block, and the word being spoken scales with the actual voice amplitude.

Requirements

Node 18+, Python 3.10+, ffmpeg, and your own Gemini API key. The free tier covers about 10 videos a day. Speech recognition and rendering run locally at no cost.

Runs on macOS, Linux, and Windows.

What this does not do

It does not edit filmed footage. There is no cutting, no filler removal, no speaker detection. It generates video from text. If you have camera footage you want trimmed and captioned, this is the wrong tool.