Medical Referral Intake Under Load: Validation, Retries, and Secure Temporary Files

Short answer: put every scanned referral behind a bounded queue, validate before OCR, retry only replayable work, and delete encrypted temporary files after a verified handoff. Under load, the winning design is the one that protects batch throughput without hiding queue age or validation failures.

I build OCR pipelines for gaming archives, where a batch can contain thousands of scanned medical referrals from tournament clinics. The domain is unusual, but the failure pattern is ordinary: a worker accepts bytes faster than OCR can process them, a timeout triggers a duplicate attempt, and a “temporary” PDF survives long enough to appear in a backup. I first thought a larger worker pool would fix this. It didn't. Queue admission and file lifetime mattered more than raw concurrency.

How should validation and retries protect referral intake latency under load?

Start with an admission contract. The HTTP edge checks content type, byte length, page-count hints, and an authenticated tenant before writing a file. It returns a job identifier quickly; OCR happens asynchronously. A malformed upload never consumes an OCR slot, and a valid upload is not held open behind another team's batch.

The queue needs two limits: maximum waiting jobs and maximum bytes in temporary storage. When either limit is reached, apply backpressure with a clear 429 and Retry-After, or route the referral to a manual intake lane. Do not accept an unbounded promise list in a Node.js handler simply because the database can store job rows. The storage budget is part of the latency budget. Keep it bounded.

Retries need an ownership rule. A network timeout means the client cannot prove whether the OCR service received the document; retrying is safe only when the job carries an idempotency key and the downstream operation honors it. A validation error is permanent and should be marked rejected immediately. A rate limit can be retried after the advertised delay. Keep a small attempt cap, then move the job to review with the original bytes and trace ID intact.

One short rule helps in code review: retry a job, never an HTTP callback.

Measure queue age, admission-to-start latency, OCR duration, retry count, bytes waiting, and the percentage of jobs that reach searchable text before the intake SLO. A single end-to-end p95 cannot tell a saturated disk from a slow recognizer. I am not sure your first threshold will be right; replay a representative batch and revise it before promising a number.

A concrete batch shape for scanned referrals

Imagine a gaming federation importing 2,400 clinic referrals after a weekend event. Each PDF has a source ID, participant ID, consent marker, and scan pages. The intake service stores a small metadata row first, then streams the upload to a private temporary directory. The worker claims jobs with a lease, runs OCR, validates extracted fields, and publishes searchable text plus a confidence report.

The lease prevents two workers from processing the same row at once, while an idempotency key prevents a timeout from creating two OCR records. Lease duration must exceed the observed OCR p99 for the largest admitted document, with a renewal heartbeat for long scans. If a worker dies, the lease expires and another worker can claim the job; the retry count still limits a poison document from circulating forever.

Validation is layered. File signatures catch a renamed executable, a PDF parser catches malformed structure, and application rules check required fields such as referral date and participant ID. OCR output then gets its own schema check: text, page count, language, and confidence bands. Keep the original scan hash beside every derived artifact so an auditor can show exactly which bytes produced the text.

The temporary file should have a random name, restrictive permissions, and an expiry independent of process cleanup. Store only a hash and job ID in logs; do not print referral text, names, or page images. On success, atomically move the encrypted artifact to its retention-controlled location, then unlink the staging file. On rejection or cancellation, unlink it immediately and record the reason without recording the medical content.

What does a queue-first Python worker actually enforce?

The following sketch is intentionally generic. It shows the state transitions and cleanup boundary; the OCR adapter can call a self-hosted engine or a managed HTTP service.

from __future__ import annotations

import hashlib
import os
import tempfile
from dataclasses import dataclass
from pathlib import Path


@dataclass(frozen=True)
class IntakeJob:
    job_id: str
    source_id: str
    content_type: str
    max_bytes: int
    attempt: int


def stage_upload(chunks: list[bytes], job: IntakeJob) -> tuple[Path, str]:
    if job.content_type != "application/pdf":
        raise ValueError("unsupported content type")

    digest = hashlib.sha256()
    total = 0
    handle = tempfile.NamedTemporaryFile(prefix="referral-", suffix=".pdf", delete=False)
    path = Path(handle.name)
    try:
        for chunk in chunks:
            total += len(chunk)
            if total > job.max_bytes:
                raise ValueError("upload exceeds byte limit")
            digest.update(chunk)
            handle.write(chunk)
        handle.flush()
        os.fsync(handle.fileno())
        os.chmod(path, 0o600)
        return path, digest.hexdigest()
    except Exception:
        path.unlink(missing_ok=True)
        raise
    finally:
        handle.close()


def process_job(job: IntakeJob, chunks: list[bytes], ocr) -> dict:
    path, source_hash = stage_upload(chunks, job)
    try:
        result = ocr(path, idempotency_key=f"referral:{job.source_id}")
        if not result["text"].strip() or result["page_count"] < 1:
            raise ValueError("ocr output failed schema validation")
        return {"job_id": job.job_id, "sha256": source_hash, "text": result["text"]}
    finally:
        path.unlink(missing_ok=True)

In production, chunks should come from a streaming request body and the queue should persist state outside the process. The important invariant is visible: every path through process_job removes the staging file, while the idempotency key stays stable across attempts. A real adapter also needs a deadline, cancellation support, and a response-size limit.

The code does not claim that OCR quality is solved by a schema check. Confidence thresholds should come from an evaluation set containing skewed scans, stamps, handwritten notes, and low-contrast copies. Keep that set versioned. Notebook-to-prod work gets safer when a prompt or parser change must pass the same cases before it reaches the queue.

Failure modes and the trade-offs behind them

The first trap is retry amplification. If 100 workers each retry three times after a 30-second timeout, the queue can grow while the upstream is recovering. In a realistic weekend import, the first wave fills the queue, the second wave is made of duplicate attempts, and the third wave contains fresh referrals that have never received a first attempt; operators looking only at completed-job count see a quiet failure because the counter excludes work still waiting. Exponential backoff with jitter, a circuit breaker, and a global retry budget protect batch throughput. A job that exhausts the budget belongs in a review queue, not in an infinite loop. The retry budget should be visible beside queue age, because a low error rate can coexist with a completely stalled intake when every worker is sleeping between attempts.

The second trap is lease expiry during a large scan. A duplicate OCR run may look like a data race, but the root cause is often a lease shorter than real processing time. Renew leases while progress is observable, and make the final write conditional on the current lease token.

The third trap is unsafe cleanup. Deleting only in the success branch leaves rejected referrals on disk; deleting before the durable handoff risks losing the source. Use an atomic handoff, verify the destination hash, and then delete staging. Your mileage may vary with the filesystem and encryption layer, so test crash recovery rather than assuming finally covers a power loss.

Delete twice if necessary.

There is a real boundary to this design. A queue-first service is not suitable when a clinician requires an interactive result inside one request; use a synchronous, size-limited path for that small workload and reserve batch intake for asynchronous jobs. Teams without operators for disk, queue, and key rotation may prefer a managed ingestion system. Teams with strict residency or custom retention rules may accept the work of running their own workers.

Do not choose on price alone. Choose the arrangement whose queue age, retry behavior, data residency, and deletion evidence you can explain during an incident review.

A small evaluation harness before rollout

Before copying the architecture, run a fixed corpus through the complete path: admission, staging, OCR, validation, durable handoff, and cleanup. Include valid PDFs, truncated files, oversized files, duplicate source IDs, and documents that should go to human review. Record job latency as a timeline, not just a final duration.

The pass criteria should be boring and explicit: no file remains after a terminal state; duplicate submissions converge on one idempotent result; permanent validation errors consume no OCR retries; queue age stays below the batch SLO at the planned arrival rate; and every searchable document retains a source hash and policy version. Test a disk-full condition and a worker crash between OCR completion and the handoff. Those are the moments that expose whether the architecture is real.