Katto already had AI dubbing. A creator could take a short clip, choose a language, and get a translated version back. The first version used Kokoro and covered eight languages.
Recently, I moved the primary speech engine to Google Cloud Text-to-Speech with Chirp 3 HD. The goal looked straightforward: improve the voices and add more languages. Katto now exposes 19 dubbing languages and 12 curated voices in the editor.
I expected the migration to be mostly an API swap.
It was not.
Generating a natural sentence is easy. Making that sentence begin and end inside the exact space left by another speaker is the actual dubbing problem.
If the original speaker says something in 2.4 seconds and the translation takes 3.2 seconds, a good TTS model does not save the clip. The voice runs into the next sentence, the captions drift, or the speech has to be accelerated until it sounds rushed.
The TTS request turned out to be the smallest part of the system.
The pipeline I ended up with
cached word timings
|
v
phrase segments with fixed time slots
|
v
length-aware translation
|
v
Google Chirp 3 HD, one phrase at a time
|
v
bounded audio alignment with FFmpeg
|
+----> translated captions from the same segments
|
v
final MP4 remux
Each stage exists because a simpler version failed in a different way.
1. Keep the original timing as data
Katto already has word timestamps from its transcription pipeline. I do not send the whole transcript to translation and TTS as one block. I group words into phrases.
A phrase ends at sentence punctuation, after a pause longer than 0.6 seconds, or when it would exceed six seconds. Every segment retains its original start and end. Those numbers become the contract for every later stage.
Synthesizing the full transcript may sound smoother, but it removes the anchors needed to put speech back on the video timeline. Phrase-level synthesis preserves enough context for a natural voice while keeping useful timing boundaries.
2. Translate for spoken duration, not only meaning
A correct written translation can be a bad dubbing translation. Some languages need more syllables to express the same idea. A literal version may preserve every nuance and still be unusable because it does not fit its slot.
For each group of phrases, I ask Claude Haiku for a natural spoken translation that prefers shorter phrasing and fewer syllables. The response preserves every source segment index.
The prompt is part of the audio system. Its job is not only linguistic accuracy. It also reduces the time stretching needed later.
This remains a heuristic. An LLM cannot guarantee duration because the selected voice also changes pacing. The audio layer still measures the generated WAV. An in-process cache keyed by target language and source segments lets repeat renders reuse the translation while the worker remains alive.
3. Synthesize one phrase at a time
Each translated phrase goes to the Google Cloud Text-to-Speech REST API as 24 kHz LINEAR16 audio. Voice names follow the locale-Chirp3-HD-speaker format documented by Google.
Google supports more Chirp 3 HD locales than Katto currently exposes. I deliberately limited the product to 19 languages wired through the complete path: UI, locale mapping, fonts, captions and fallback behavior. An API claiming a language does not mean the surrounding product supports it correctly.
4. Fit speech into the slot without destroying it
After synthesis, I compare the generated duration with the original slot.
target_duration = segment["end"] - segment["start"]
ratio = generated_duration / target_duration
If the speech is slightly too long, FFmpeg's atempo filter speeds it up while preserving pitch. I cap the speedup at 1.3x.
That cap matters. An earlier 1.6x limit made more phrases fit, but some voices sounded unnaturally hurried. Perfect alignment is not useful if the result is unpleasant to hear.
Each adjusted phrase is placed at its original offset in a master track. Short phrases leave a natural pause. Longer phrases get a bounded correction. This does not solve every pathological translation. It enforces a more useful rule: preserve intelligibility before chasing mathematically perfect timing.
5. Give speech and captions the same clock
The synthesis response gives me audio, not reliable word timestamps for the newly spoken text. Inventing word-level timing would make the captions look precise while being wrong.
Dubbed clips therefore use phrase-level captions generated from the same translated segments used for speech. Their start and end times already match the audio placement.
The renderer also selects fonts that cover the target script. Latin fonts are not enough for Japanese, Chinese, Hindi or Arabic. Intro titles and AI hooks follow the same localization path, so the screen does not speak one language and display another.
The ordinary production problems mattered too
Not every curated speaker worked in every locale I tested. If a selected voice is unavailable, Katto retries once with a known default for the chosen gender. The editor's live preview calls the same Google engine. A sample from another model would let the user choose a voice they never receive.
The production API key is restricted to the VPS IPv4 address, but the VPS preferred IPv6 when reaching Google. Valid requests were rejected until the TTS call was forced through IPv4. The API, key and code were correct. The network path was not.
Dubbing produces a separate video variant. If translation, synthesis, caption rendering or remuxing fails, the original clip remains intact. For the original eight languages, Kokoro remains available when the Google path is disabled or unconfigured.
What changed, and what did not
The visible result was an expansion from 8 to 19 dubbing languages, plus selectable voices and live previews.
The deeper improvement was architectural. Translation, speech, captions, overlays and video rendering now share one temporal unit: the phrase segment.
The system does not clone the original speaker, perform lip sync or invent generated word timestamps. Very long translations can still exceed their ideal slot. Those limits are better than claiming precision the pipeline cannot deliver.
I thought I was replacing a speech engine. I ended up tightening the contract between almost every stage of the media pipeline.
The API call was the easy part. Making the result belong on the original timeline was the work.
Katto turns long videos into short vertical clips and can now dub those clips into 19 languages. If you are building multilingual video, I would be interested to hear how you handle timing without making the voice sound rushed.
