Short answer: inspect metadata and preserve the original mobile photo before changing pixels; then normalize rotation once, treat crop as a reversible proposal, and accept the result only when the full receipt remains visible. That order gives a scanned receipt expense app a clean image for extraction without throwing away evidence it may need later.
The hard part isn't drawing a rectangle. It is deciding which transformations are facts, which are guesses, and which can destroy useful content. I care about time-to-first-call, but a tiny API that silently rotates twice or clips a tax line is bad developer experience. The contract needs to make those states boring and observable.
No config maze.
How should receipt capture metadata guide rotation and crop choices?
Use metadata as input evidence, not as the final truth. A mobile upload arrives as encoded bytes plus whatever descriptive fields survived capture, sharing, or browser conversion. The intake stage should record what it can inspect, ask a decoder for the displayed orientation, and keep those observations beside the immutable source. It should not mutate the source while it is still deciding what the image means.
The first durable output is a normalized derivative whose pixels have one declared orientation. Every later stage reads that derivative and ignores the source orientation hint. This rule prevents an easy integration failure: one layer honors an orientation hint during decode, another sees the copied hint and applies it again. The failure can look plausible in a thumbnail, which is exactly why I don't trust visual spot checks alone.
Crop comes later because it is less certain. Rotation can be expressed as a discrete normalization decision. Receipt boundaries are an estimate made from contrast, geometry, or a detector. Shadows, folded corners, a pale receipt on a white table, and handwritten tips all make the estimate less trustworthy. Store the proposed rectangle in normalized coordinates, along with the decision that produced it, before rendering another derivative.
The invariant is simple: source bytes never change; each accepted transform names its input; and extraction, preview, and audit views agree on the derivative identifier they consumed. That costs a little storage and saves a lot of forensic guessing.
Build the smallest pipeline with explicit artifacts
I would start with one function and three artifacts: the source, the orientation-normalized image, and an optional cropped image. The image operations below are intentionally injected. The contract matters more than binding the article to a particular decoder or edge detector.
interface StoredImage {
id: string;
width: number;
height: number;
}
interface IntakeMetadata {
mediaType: string;
displayedQuarterTurns: 0 | 1 | 2 | 3;
}
interface CropProposal {
x: number;
y: number;
width: number;
height: number;
confidence: number;
}
interface ImageOps {
inspect(bytes: Uint8Array): Promise<IntakeMetadata>;
storeOriginal(bytes: Uint8Array, mediaType: string): Promise<StoredImage>;
normalizeOrientation(sourceId: string, turns: 0 | 1 | 2 | 3): Promise<StoredImage>;
proposeReceiptCrop(imageId: string): Promise<CropProposal | null>;
renderCrop(imageId: string, crop: CropProposal): Promise<StoredImage>;
}
interface ReceiptAsset {
source: StoredImage;
normalized: StoredImage;
cropProposal: CropProposal | null;
extractionInput: StoredImage;
}
function cropIsUsable(crop: CropProposal, minimumConfidence: number): boolean {
const insideFrame =
crop.x >= 0 &&
crop.y >= 0 &&
crop.x + crop.width <= 1 &&
crop.y + crop.height <= 1;
return insideFrame && crop.width > 0 && crop.height > 0 && crop.confidence >= minimumConfidence;
}
async function ingestReceiptPhoto(
bytes: Uint8Array,
ops: ImageOps,
minimumCropConfidence: number
): Promise<ReceiptAsset> {
const metadata = await ops.inspect(bytes);
const source = await ops.storeOriginal(bytes, metadata.mediaType);
const normalized = await ops.normalizeOrientation(
source.id,
metadata.displayedQuarterTurns
);
const cropProposal = await ops.proposeReceiptCrop(normalized.id);
const extractionInput =
cropProposal && cropIsUsable(cropProposal, minimumCropConfidence)
? await ops.renderCrop(normalized.id, cropProposal)
: normalized;
return { source, normalized, cropProposal, extractionInput };
}
The useful detail is the fallback. A missing or weak crop proposal does not block capture; the normalized full frame becomes the extraction input. The threshold is application policy, so it belongs in configuration, but it should be one named value rather than a nest of device-specific flags. Start with evidence from your own validation set. I'm not sure a universal threshold exists, because receipt layouts and capture conditions vary; a labeled corpus from the actual app is what would settle it.
I would also reject a decoded image that cannot yield valid dimensions before any detector runs. That is an input boundary, not a retry strategy. Keep client-facing failures compact and stable, while internal logs retain the stage and artifact identifier. Don't leak decoder chatter into the mobile UI.
Test decisions, not pretty thumbnails
A screenshot gallery is useful for humans, but it is a weak regression suite. Benchmark the pipeline as a sequence of decisions. Feed it fixtures representing portrait and landscape captures, absent inspection fields, large borders, tight framing, low-contrast edges, and a receipt that already fills the frame. Assert which artifact becomes extractionInput, not just that some JPEG-shaped output exists.
One fixture should be deliberately asymmetric: put a marker near the original top-left and readable labels along two edges. After normalization, the test can verify both location and dimensions. A symmetric receipt outline can hide a half-turn error. This is the sort of test that looks fussy until two independent image layers each try to be helpful.
interface FixtureExpectation {
name: string;
expectedTurns: 0 | 1 | 2 | 3;
shouldCrop: boolean;
}
const cases: FixtureExpectation[] = [
{ name: "portrait-with-border", expectedTurns: 0, shouldCrop: true },
{ name: "landscape-capture", expectedTurns: 1, shouldCrop: true },
{ name: "tight-frame", expectedTurns: 0, shouldCrop: false },
{ name: "uncertain-edge", expectedTurns: 0, shouldCrop: false }
];
for (const fixture of cases) {
// Resolve fixture bytes and expected geometry from the repository test data.
void fixture;
}
Those expectations are illustrative fixture labels, not benchmark results. Record real timings separately for inspection, decode and orientation normalization, crop proposal, crop rendering, and downstream extraction. Percentiles matter more than one warm local run, and peak memory deserves its own measurement because mobile photos expand substantially when decoded. Publish no performance claim until the harness, hardware, input set, and concurrency are named.
Observability should follow the same stage boundaries. Count how often the system uses the normalized fallback, how often a human adjusts the proposed crop, and how often extraction is retried with the full frame. Avoid logging raw receipt pixels or extracted text as routine diagnostics. Artifact IDs, dimensions, decision reasons, and timings are enough to debug the pipeline shape without turning logs into another copy of sensitive expense data.
Short version: measure the branch choices.
What I would change at scale, and where this design loses
At higher volume, I would make normalization and crop rendering idempotent jobs keyed by source identity plus transform version. That makes retries predictable and allows a detector upgrade to produce a new proposal without rewriting old evidence. I would also separate interactive preview latency from background extraction throughput; they use the same artifacts, but they do not need the same queue or deadline.
The catch is extra storage and lifecycle bookkeeping. Keeping a source plus one or two derivatives is not suitable when policy requires immediate deletion of originals, when the application is strictly offline, or when a review workflow has no legitimate need for retained evidence. In those cases, keep the same ordered state machine in memory, emit only the permitted final artifact, and preserve a small decision record without the image. Stick with full-frame normalization when crop mistakes cost more than the downstream bandwidth or extraction noise. Use automatic cropping only after the adjustment rate on representative receipts is acceptable to the team that owns the expense workflow.
There is another trade-off: a generic ImageOps boundary makes decoder replacement cheap, but it can conceal capability differences. Keep the interface narrow, then expose factual outputs such as dimensions, media type, orientation decision, and crop proposal. Don't grow an option for every library switch. Config bloat is still coupling; it just moved into a file.
The final selection rule is not a brand ranking. Choose an image stack that can inspect the media types your clients actually upload, normalize orientation exactly once, render from explicit coordinates, and expose enough timing and memory data to benchmark the deployed path. If one candidate cannot preserve the original while producing traceable derivatives, it is a poor fit for receipt evidence. If another requires an SDK-specific object to cross every application boundary, wrap it at ImageOps or keep looking.
References
Media containers and browser support differ, so accepted upload types should be an explicit product decision rather than an accidental decoder default. The MDN media formats guide is the reference used here for that boundary.
