More Than Subtitle Translation: Thoughts on Building a Video Localization Pipeline

Background and Goal

When I first started working on subtitle localization, I naturally thought of it as a translation problem: take the English subtitles, turn them into Chinese, smooth out the wording a bit, and publish.

After processing a few videos in a row, that idea fell apart pretty quickly. The problems were not only in the translation. Often, Chinese subtitles were hard to read because the English source already had noise. Terminology drifted because every video was reinventing translations from scratch. Titles missed the point because the full content had not really been understood. Covers looked plausible, but did not quite match what the video was actually about.

I gradually realized that video localization should not be treated as a single translation call. It is closer to a content production pipeline: first turn the source into reliable material, then convert it into the target language, and finally package that content into something publishable, searchable, and worth clicking.

This article is a summary of the thinking behind that pipeline. It is not a user guide for a specific tool. It is a set of judgments I developed while working on video localization: what should be separated, which intermediate artifacts are worth keeping, what kind of feedback should be retained, and why “continuous iteration” matters more than “getting this one translation done.”

Overall Architecture

I now prefer to break video localization into several connected stages, each with a different responsibility:

Rendering diagram...

The important part of this diagram is not the tooling. It is the boundaries.

Is the source reliable? Are the subtitle segments readable? Does the Chinese read naturally? Is the terminology stable? Does the title capture the video’s main point? Does the cover express the right hook? These questions are often hidden inside the word “translation,” but they are not the same kind of problem. They should not be solved by one monolithic step.

I roughly think of the pipeline in three layers:

• Content layer: turn the video into a reliable, clear source.

• Language layer: convert that source into natural, accurate Chinese that works as subtitles.

• Publishing layer: produce the title, chapters, introduction, keywords, and cover.

If you stop at the language layer, you usually only have subtitles. Once you reach the publishing layer, the video is much closer to being truly localized.

If we lay the stages out, each one is solving a different problem:

This table may look a little “engineering-heavy,” but it helps me avoid a common trap: dumping every problem into the translation stage.

Why Not Just Use an End-to-End Agent?

A natural question is: if models are getting stronger, why not hand the video to an end-to-end agent and let it generate subtitles, titles, and covers automatically?

I understand the appeal. I have tried similar approaches. They look simple, the path is short, and the first demo can be surprisingly impressive. But once you move into repeated production, the biggest problem is control.

When the final output is poor, it becomes hard to tell where things went wrong. Did ASR mishear something? Was the source not cleaned properly? Did translation drift? Was terminology inconsistent? Did the title misunderstand the point? Did the cover choose the wrong hook? If all of these problems are wrapped inside one generation step, the system has very little room to learn.

The value of a pipeline is not complexity for its own sake. The value is observability. Each stage has its own input and output, which means it can be checked, rerun, and improved on its own.

I increasingly think of intermediate artifacts not as extra files, but as debugging information for the pipeline. Without them, all you can see is whether the final subtitles are good or bad. With them, you can see where the result started to go wrong.

Pipeline Details

Source Acquisition

The first step in localization is not translation. It is understanding where the source comes from.

The source may come from official subtitles, platform-generated subtitles, or ASR. All three can become an SRT file, but their reliability is very different. Official subtitles are usually more stable. Auto subtitles often have strange segmentation. ASR is useful when there are no subtitles, but names, terminology, punctuation, and sentence boundaries can all be wrong.

At this stage, I want to do two things: unify the entry point, and preserve provenance.

Unifying the entry point means downstream stages do not need to care where the subtitles came from. Preserving provenance means later decisions have context. A human-authored subtitle track and an ASR draft should not be treated the same way. The former may only need light cleanup; the latter requires a much stronger assumption that noise exists.

Subtitle Extraction / ASR

ASR is a good entry point, but a bad endpoint.

It solves the problem of how to get started when no subtitles exist. Once video speech has been converted into a draft, the pipeline has material to work with. But that draft should not be treated as truth. This matters especially for technical videos: if a product name, command-line flag, or API name is misheard, translation will often amplify the mistake naturally.

That is why I do not like the “ASR, then translate immediately” approach. It saves a step, but it makes errors more hidden. By the time the Chinese text comes out, it may sound fluent, even though the source was already wrong.

ASR should be followed by cleaning. The step is not glamorous, but it determines the foundation every later stage stands on.

Subtitle Cleaning

Subtitle cleaning is a stage I have come to value more and more.

At first, I also wondered why English subtitles needed a dedicated cleanup pass when models were already good at translation. Later I realized that many “translation problems” are not caused by Chinese generation at all. If the source has misrecognitions, repetitions, broken phrasing, or unstable spellings, Chinese proofreading can only patch the damage.

Cleaning is a modest but important job: fix obvious ASR noise, normalize punctuation and casing, standardize terminology and names, and remove speech artifacts or repetitions that should not enter the translation stage. It does not translate. It should not rewrite the subtitle structure. It simply turns the source into a more reliable baseline.

I think of it as upstream quality control. Do not wait until the end and force Chinese proofreading to carry every problem. The earlier source noise is handled, the easier every later step becomes.

This is also an important tradeoff in a pipeline: every stage needs boundaries. Cleaning cleans. Segmentation segments. Translation translates. When the boundaries are clear, you know where to go back when something breaks.

Source Segmentation

Subtitles are not prose. They are a timed reading experience.

Auto subtitles often split a complete sentence into awkward fragments. Each cue may be technically correct on its own, but reading them in sequence can feel like assembling a puzzle. Chinese is especially sensitive to this, because half-sentences that English can still tolerate with word order and prepositions often feel broken in Chinese.

So before translation, I prefer to segment the source into units that are easier to read and translate. This stage does not rewrite the source. It only merges and splits: join fragments that are too small, split cues that are too long, and avoid leaving half a thought stranded in a single subtitle.

The goal is not cosmetic neatness. The goal is to give the translation stage fuller semantic units. Better source segmentation directly reduces awkward Chinese phrasing.

Translation

The translation stage should stay relatively simple.

Its job is to produce a Chinese draft from a reliable source. It should not also be responsible for final polishing, terminology governance, title strategy, or chapter planning. The more responsibilities are mixed together, the harder it becomes to judge what went well and what failed.

Keeping translation separate has a practical benefit: the source baseline is preserved, and so is the Chinese draft. If you later want to change models, adjust style, or rerun proofreading, you do not need to redo the upstream source work.

An all-in-one pass looks fast, but it often turns the problem into a black box. A pipeline may look more roundabout, but each layer leaves behind an artifact, which gives you a place to improve.

Proofread

A Chinese draft is not the final subtitle.

Translation models usually focus on whether meaning has been transferred. Publishable subtitles need more: Does this line read naturally in context? Is terminology consistent with previous cues? Can this cue be read within its duration? Did segmentation leave a phrase hanging? Is there a fluent-sounding line that actually deviates from the source?

These questions belong in the proofread stage.

Proofreading, as I understand it, is not simple polishing. It is a comparison: treat the source as the factual authority, treat the Chinese draft as text to improve, and use surrounding context to decide whether it is truly publishable.

One boundary is especially important: reference Chinese subtitles should remain references. They should not take over. Platform subtitles may read smoothly, or they may be machine-translated. Even human subtitles can omit or adapt content. The final authority should still be the source.

Proofread is the step that moves the work from “understandable” to “publishable.” It handles viewer experience, and it is the part of the pipeline closest to a human subtitle editor.

Glossary

If you only translate one video, a glossary can feel unnecessary. Once you do this for a while, it becomes one of the most valuable assets.

Across a channel, if the same concept is kept in English today, translated tomorrow, and phrased differently the day after, it may not ruin any single video. But over time, the content system starts to feel loose. Viewers may even wonder whether these are the same thing.

Glossary is not about one-off translation. It is about long-term consistency.

But glossaries can also go wrong in the other direction: everything gets added. Brand names, ordinary nouns, temporary phrases, abbreviations without enough context—all of them enter the glossary. At that point, it stops being a quality asset and becomes prompt noise.

I prefer a small and precise glossary. New videos can produce candidate terms, those candidates can be normalized, and ambiguity can be flagged. But before something enters the formal glossary, it should be reviewed. Discovery can be automated; approval should remain conservative.

The value of a glossary is not collecting as much as possible. It is stabilizing the expressions that truly need to remain consistent over time.

Metadata Generation

When subtitles are finished, the video is still not fully ready to publish.

Titles, keywords, chapters, introductions, and source attribution determine whether viewers want to click and whether the content can be found later. They are not accessories to the subtitles. They are part of the localized content.

I prefer to think of metadata generation as content understanding. It should not simply translate the original title, and it should not summarize only the first few minutes. It needs to look at the full content and identify the real throughline: What change is being discussed? What problem is being solved? What should viewers remember?

A good title should be specific, not a generic phrase like “major release” or “new upgrade.” Good chapters should help people scan the content, not slice the video mechanically every few minutes. A good introduction should cover the whole video, not restate the title. Source attribution is simpler: if you know it, say it; if you do not, admit that you do not.

When this layer is done well, a video moves from “subtitled” to “publishable.”

Thumbnail Generation

A thumbnail may look like visual design, but it is also content judgment.

A good thumbnail first needs to know the core value of the video. Is it a new capability? A conflict? A result? An unexpected change? If the subtitles and metadata have not clarified that, the thumbnail can easily become pretty but empty.

That is why I put thumbnail generation near the end. First subtitles, then metadata, then a cover based on the content’s main storyline. The headline, visual evidence, brand presence, and aspect-ratio composition should all serve the same message.

I care especially about one thing: do not invent things for the sake of making the cover look good. If there is no real UI, do not draw a fake UI. If there is no usable speaker image, do not pretend there is a live photo. A cover can be abstract, but it should not mislead.

The thumbnail is not decoration. It is the final piece of the publishing expression.

Feedback and Learning Loop

If the pipeline processes each video and leaves all experience in someone’s head, it will not really improve.

Feedback should be retained, and ideally it should return to the right place. If subtitles are too literal, that should influence future proofreading. If titles miss the point, that should influence metadata. If thumbnails become unreadable when scaled down, that should influence thumbnail generation. If a term keeps getting corrected, it should enter glossary review.

I do not like collapsing all feedback into “improve quality.” It sounds right, but it is not actionable. Useful feedback should have a location and a type: which subtitle cue, which term, which title strategy, which cover problem. The more specific it is, the easier it is to preserve.

At the same time, feedback should not become a new doctrine. Historical experience can support judgment, but it cannot override the current video. A title pattern that worked last time should not automatically be reused. A term that is usually kept in English does not mean it can never be translated in any context.

If glossary is long-term memory for terminology, feedback is long-term memory for quality. It helps the pipeline do more than repeat itself. It helps the system develop preferences, avoid old mistakes, and retain what worked well.

That is the most interesting part to me: the sign of a mature localization system is not how fast the first run is. It is whether the tenth or fiftieth run becomes more stable.

Engineering Optimizations

If you only translate a video once in a while, a manual workflow is enough. Once you want to do this consistently, engineering problems appear quickly: What happens when something fails? How do you rerun a local change? How do you deal with unstable model output? How do you control cost? How do you preserve quality learning?

Batch Processing

Subtitles are well suited to batch processing.

A single cue is too short, and the model lacks context. A whole video at once is too large, too expensive, and hard to recover from when it fails. Batching is a compromise: give the model enough context while limiting the failure scope to a small section.

Many LLM workflows share this problem. Do not only ask, “Can the model do this in one pass?” Also ask, “If it does not work, can I rerun only the broken part?”

Concurrency Control

Concurrency can improve speed, but subtitle work cannot be optimized only for throughput.

A video is a continuous narrative. Batches can influence each other, especially in terminology, tone, and contextual continuity. Too much concurrency may save time while making the result less stable. In localization, stability itself is part of quality.

So I prefer controlled concurrency: know which stages can run in parallel, and which ones should stay conservative.

Retry / Backoff

A mature pipeline cannot assume the world will behave.

Downloads fail. Models time out. A batch returns incomplete output. Subtitle formatting occasionally breaks. A script usually stops at that point; a system should try to determine whether this is a temporary failure, a local anomaly, or something that requires human intervention.

Temporary failures can be retried. Repeated failures can back off. Local anomalies can be split into smaller chunks. Format errors can be traced to a specific cue. This prevents a small issue from collapsing the entire workflow.

Cache

Caching looks like a way to save money and time, but I think its larger value is supporting iteration.

Subtitle work often involves repeated adjustments: change a term, fix a few lines, rewrite the title, try another cover. If every small change requires a full rerun, people become less willing to experiment. A good cache makes iteration feel safe, because you do not pay the full cost for every small correction.

Of course, the cache must be trustworthy. If inputs change, rules change, the glossary changes, or model configuration changes, old results may no longer be valid. The hard part is not storing cached data. The hard part is knowing when it should expire.

Error Recovery

I increasingly think the most important part of a pipeline is not the happy path, but the recovery path.

Subtitle work has many small failures: a cue has bad formatting, a Chinese line is too dense, a term is mishandled, a batch returns one item short. If the system can only fail the whole video, maintenance becomes expensive.

The better approach is to localize errors as much as possible: stage, batch, cue. The more precise the location, the lighter the fix. Human intervention also becomes easier, because the person is facing a concrete issue rather than a failed video.

File Organization

When maintaining a pipeline over time, directory structure matters more than it first appears.

Source acquisition, generation stages, validators, glossary, orchestration, and shared utilities should each have a clear home. When a stage needs to change, it should not shake the entire system.

I do not see this as “engineering for engineering’s sake.” File boundaries are thinking boundaries. When the boundaries are clear, the system can keep growing instead of slowly becoming one giant script nobody wants to touch.

Current Limits and Future Improvements

There is still a lot to improve in this kind of pipeline.

Multilingual source support is one obvious direction. Different languages have different subtitle rhythms, segmentation habits, and cleanup priorities. One generic rule set will not be enough forever.

Term discovery can also become smarter. Models can help explain candidate terms, identify cross-sentence entities, and judge whether an abbreviation is truly worth retaining. But I still do not trust fully automatic glossary approval. Once long-term memory is polluted, cleaning it up is much more painful than adding terms carefully.

Human review is another area worth productizing. Candidate terms, low-quality subtitles, title versions, and cover versions could all be gathered into a review workflow. Human judgment would become more stable, and feedback would flow back more easily.

Quality evaluation can also become more systematic. Format validation is only the baseline. The harder problems are omissions, terminology coverage, subtitle density, source-target consistency, and contextual coherence. These can gradually become automatic checks or semi-automatic warnings.

Thumbnail feedback is already part of the loop, but it can be made more fine-grained. A cover is not finished once an image is generated. The system can keep learning what kind of headline is clearer, what kind of visual evidence is more credible, and what kind of composition remains readable at small sizes. The next step is to summarize and evaluate that feedback more systematically.

All of these improvements return to the same question: can the system learn from its results?

Conclusion

I no longer think of video localization primarily as “subtitle translation.” Subtitles are only one artifact.

The fuller goal is to turn a raw video into publishable content that another-language audience can understand smoothly, want to click, find later, and trust. That involves language transfer, but also content understanding, terminology management, quality feedback, and visual expression.

Model capability matters, of course. But what matters more is placing the model inside the right boundaries. ASR opens the door. Cleaning establishes a reliable source. Segmentation improves the reading structure. Translation produces the draft. Proofread controls quality. Glossary preserves long-term memory. Metadata and thumbnail complete the publishing expression.

Once these stages are separated clearly, localization is no longer a one-off act of “moving text into another language.” It becomes a content system that can keep iterating, accumulating, and improving.