Every "translate your PDF" tool works beautifully until you feed it a scan.
Then you get back a wall of reflowed text. The table is gone. The stamp is floating over a paragraph. The two-column layout became one column. For a digital PDF this doesn't happen, because the text is in the file and you can swap it in place. For a scan there is no text — there are pixels that look like text.
I spent a while building a pipeline for this, and the thing that surprised me is how little of the work is translation. Translation is one API call. The hard parts are all graphics:
- Erase the original ink from the bitmap without destroying what's underneath.
- Re-typeset the translation into the space it left — usually more text than you started with.
- Do both fast enough that a long document isn't a coffee break.
Here's roughly what each of those involves.
1. The mask is the whole game
To erase text you need an inpainting model (I use LaMa) and a binary mask saying which pixels to remove. The mask is where all the quality lives.
The obvious mask is the OCR bounding box, filled solid. Don't do this. Official documents are exactly the case where the box is full of things you want to keep: table rules, the underline of a signature line, the edge of a seal, background texture. Fill the box and you erase all of it, and the inpainter cheerfully hallucinates blank paper in its place.
What works better is masking only the ink. Otsu-threshold inside the box, and take the dark class:
roi = cv2.cvtColor(img_bgr[y0:y1, x0:x1], cv2.COLOR_BGR2GRAY)
_, strokes = cv2.threshold(roi, 0, 255,
cv2.THRESH_BINARY_INV + cv2.THRESH_OTSU)
This breaks on white-on-black headers — Otsu has no idea which class is ink. The fix is cheap and has been reliable: the border of a text bbox is almost always background, so if the border mostly landed in the "stroke" class, flip it.
border = np.concatenate(
[strokes[0, :], strokes[-1, :], strokes[:, 0], strokes[:, -1]]
)
if border.mean() > 127: # light text on dark background
strokes = 255 - strokes
Then dilate a little so anti-aliased edges get taken too. Leftover anti-aliasing shows up as a grey ghost of the original word, which looks worse than a slightly-too-large mask.
One more thing worth doing: don't send the whole page to the GPU. Merge nearby boxes, crop each group with a context margin, and inpaint the crops. LaMa needs surrounding context to reconstruct texture, but it needs local context — a modest margin does as well as the full page, and you move a fraction of the bytes.
2. The translation doesn't fit
German is long. So is Spanish, relative to English. You will routinely get a translated paragraph 30% longer than the source, and it has to go back into a box of fixed size.
You also lost the line breaks — OCR gives you text, not the original wrapping, so you have to re-wrap, and the wrap depends on the size you pick. It's a 2D fit with no closed form. Binary search:
def max_fit_scale(item) -> float:
if fits(item, 1.0):
return 1.0
lo, hi = FLOOR, 1.0
if not fits(item, lo):
return lo # even the floor overflows
for _ in range(12):
mid = (lo + hi) / 2
if fits(item, mid):
lo = mid
else:
hi = mid
return lo
fits() is a dry run of the real wrap on a scratch page, so the test and the render can't disagree. That sounds obvious and I did not do it that way at first; having a cheap approximate fits() produced a class of bug where the fit search was confident and the renderer overflowed anyway.
Two things I'd pass on:
Shrink per paragraph, not per line. If each line finds its own best scale, a paragraph where line 3 happens to be dense renders line 3 smaller than lines 2 and 4. Every line individually fits, and it reads as broken.
Keep a floor and treat hitting it as a signal, not a fallback. A region that can't fit even at half size usually isn't a fitting problem — it's OCR having merged two blocks upstream. Shrinking to unreadability hides the actual bug.
3. Making it not slow
Per page the stages are: analyze (local, fast), translate (LLM API), inpaint (GPU API), render (local, fast).
The useful observation is that translate and inpaint touch completely different inputs — one needs text, the other needs pixels — so they have no reason to be sequential:
analyze
├─ translate (API) ─┐ independent: text vs pixels
└─ inpaint (GPU) ─┘ run concurrently
render
Page latency becomes analyze + max(translate, inpaint) + render instead of the sum. Both middle stages are network-bound, so this is close to free.
You can push it further. Translation only needs the OCR text, which exists before you rasterise anything — so translating before the page bitmap is in memory means the memory-heavy pixel stage never sits blocked on a network call. On a long document that's the difference between comfortable memory use and an OOM.
One LLM-engineering note while I'm here: batch whole paragraphs into one request with index-tagged segments in and strict JSON out.
src = [{"i": i, "t": t} for i, t in enumerate(texts)]
Indices in the payload, not just array order. If the model drops a segment you want that one paragraph to fall back to its source text — not every subsequent paragraph to shift up by one and land in the wrong box. That failure mode is silent, and it is genuinely miserable to debug from a rendered PDF.
What makes this problem domain hard
If you're considering building in this space, the difficulty is not where it looks like it is. Some things that are harder than they appear, none of them specific to my implementation:
- OCR errors are unrecoverable downstream. If OCR merges two table rows because the separator is faint, no amount of good rendering fixes it. Quality is capped upstream, and most of what feels like a "layout bug" is an OCR bug.
- Writing systems are not interchangeable. Vertical Japanese needs a different layout model, not a rotated one. Arabic needs cursive joining and RTL to survive three format conversions in a row. Every script you add is real work, not a font swap.
- Every failure is visual. You cannot unit-test "this looks right." Building a way to eyeball page-level diffs quickly matters more than it should.
The general lesson, if there is one: for scanned documents, translation quality is rarely the bottleneck. Getting the ink off the page cleanly, and putting new ink back so that it belongs there, is where the perceived quality comes from. A mediocre translation in the right place looks better than a great translation in a wall of reflowed text.
The pipeline described here runs at tryreglyph.com if you want to throw a scan at it. Happy to go deeper in the comments — the masking step in particular, I'd like to hear how other people have approached it.
