Short answer: put contract signing behind an idempotent asynchronous job, validate the manifest before any expensive work, and keep encrypted temporary files on a bounded local volume; measure queue wait and signing latency separately.
A marketplace case file is rarely one PDF. It is a packet of evidence, counterparty data, and a contract that must be signed server-side with an audit trail. Under load, the expensive part is not accepting the request. It is moving a large set of bytes through validation, canonicalization, signing, and durable storage without losing the chain of custody.
The decision record below assumes a Node.js service receives a manifest and object references, then emits a signed artifact and immutable audit events. Batch throughput is the primary axis. User-facing latency still matters, but the API should acknowledge a valid job quickly and report progress from durable state.
Invariants and failure boundaries
The manifest is the unit of identity. Compute an idempotency key from the case ID, document digests, signer identities, and signing policy version. A retry with the same key must return the original job, never create a second signature. Store each state transition with an event ID, actor, timestamp, input digest, and policy version; an audit row that only says “done” is not an audit trail.
Validation has two layers. Cheap checks run synchronously: schema, size limits, allowed media types, and that every referenced object belongs to the case. Deep checks run in the worker: PDF parsing, malware scanning, and signature-policy validation. Rejecting a malformed manifest before enqueueing protects queue capacity. Rejecting a corrupt attachment in the worker protects the signing key.
Retries need a boundary. Retry network reads, object-store timeouts, and temporary capacity responses with exponential backoff and jitter. Do not retry a deterministic schema failure or a policy decision. A lease with an expiry lets another worker reclaim a job after a process exit; an attempt number and idempotency key make that takeover observable.
The catch is that asynchronous processing is unsuitable when a caller truly needs a signed response inside one transaction, such as a regulated kiosk with a hard two-second interaction limit. Keep a small synchronous path for a pre-validated, single-document case, and stick with a transaction-local signer when the business operation cannot be split.
How should Node.js services handle asynchronous jobs, retries, validation, secure temporary files, and latency under load?
Treat the queue as a pressure valve, not as a second database. Persist the job record and its idempotency key before publishing a notification; workers can safely poll or consume the notification because the record is the source of truth. Cap concurrency by memory and signing-key access, not by CPU count alone. A 200 MB case multiplied by 20 workers is already four gigabytes before parser overhead.
Temporary files deserve the same threat model as the final document. Create them with exclusive permissions, random names, and a directory that is not served by the web process. Stream downloads into a quota-aware file, verify the expected digest, and unlink in a finally block. For very large inputs, pass a file descriptor to the parser instead of buffering a Blob-sized payload in JavaScript heap. The MDN Blob model is useful at an API boundary, but a server worker should make its memory budget explicit. In practice, reserve headroom for parser copies, digest buffers, and the runtime itself; otherwise a burst of valid jobs can trigger garbage-collection pauses that look like random signing latency. Record the temporary directory's free bytes alongside queue depth, because a full volume is a capacity failure even when CPU is idle.
Here is the critical path in Python-like pseudocode. The names are deliberately generic so the same controls can sit over any queue and object store.
async def run_case(job):
if not await jobs.claim(job.id, lease_seconds=90):
return
temp_paths = []
try:
manifest = await cases.read_manifest(job.case_id)
validate_manifest(manifest)
key = idempotency_key(manifest, job.policy_version)
if await signatures.exists(key):
return await jobs.finish(job.id, await signatures.result(key))
for item in manifest.items:
path = await secure_temp.download(item.object_ref, max_bytes=item.size)
verify_digest(path, item.sha256)
temp_paths.append(path)
await validate_attachment(path, item.media_type)
artifact = await signer.sign(temp_paths, manifest.signers, job.policy_version)
result = await artifacts.store(artifact, case_id=job.case_id)
await audit.append("signed", job.case_id, key, result.digest)
await signatures.put_once(key, result)
await jobs.finish(job.id, result)
except RetryableError as exc:
await jobs.retry(job.id, delay=backoff(job.attempt), reason=str(exc))
except PermanentError as exc:
await audit.append("rejected", job.case_id, job.id, code=exc.code)
await jobs.fail(job.id, code=exc.code)
finally:
for path in temp_paths:
secure_temp.remove(path)
Small queues lie.
Measure three clocks: admission latency (request to queued), queue wait (queued to claimed), and processing latency (claimed to completed). A single p95 “API latency” hides the queue that is actually hurting customers. Track bytes per job, retry counts by reason, lease expirations, validation rejection codes, and temporary-volume utilization. Alert on age of the oldest queued job and on a rising ratio of reclaimed leases.
Option comparison for batch throughput
| Option | Throughput behavior | Audit and retry shape | Best fit |
|---|---|---|---|
| Synchronous request/response | Ties connection and heap to document size; collapses under bursts | Hard to resume safely; partial failures are awkward | Tiny, pre-validated files with a strict interactive deadline |
| In-process background tasks | Fast to prototype, but deploys and crashes erase work unless state is external | Requires custom durable state and recovery | Low-volume internal tooling |
| Durable queue plus workers | Bounded concurrency and independent horizontal scaling | Explicit leases, idempotency, and event history | Large marketplace batches and bursty demand |
| Workflow engine | Strong timers and compensation, with operational overhead | Rich state model, more components to govern | Multi-party signing with long waits and approvals |
The durable queue wins this scenario because it makes backpressure visible and lets signing workers scale independently from HTTP handlers. It is not free: operating a broker, dead-letter policy, and replay tooling adds work. If your team cannot run those controls, a managed workflow may be safer than a homegrown queue, even if its per-step overhead is higher.
The rejected shortcut and its valid use
I would reject “upload everything, then sign in one handler.” It looks simple until a client disconnects after the upload, a retry signs the same contract twice, or a parser allocates several copies of a large buffer. Those failures turn latency into data-integrity incidents.
The shortcut is valid for a bounded internal tool where files are small, the signer is local, and losing an in-flight request is acceptable. Put a hard byte limit on it and keep the same manifest validation and audit event format, so moving to workers later does not create a second compliance model.
One uncertainty remains: the right worker concurrency depends on parser memory, key-service limits, and the storage provider's tail latency. Your mileage may vary. Load-test with realistic case sizes and inject timeouts before choosing a number; a neat benchmark on 1 MB fixtures proves very little about a 200 MB packet.
