Logistics Retention Control: A Daily Cleanup Job Deletes Old Uploads and Records

Short answer: run a daily cleanup job outside the Express process to delete old uploads, logs, and records, but enqueue bounded batches so retention work cannot consume a rate-limited logistics worker pool.

The least complex option is system cron when one host already has clear ownership, durable disks, and ordinary host monitoring. Choose a managed scheduler when host replacement or multiple application replicas could create ambiguous ownership. Choose a durable queue when cleanup can exceed its window, needs controlled concurrency, or must resume from acknowledged batches. The cron expression is the easy part; for a 02:17 UTC run it is 17 2 * * *. Capacity isolation is the decision.

This matters in logistics because proof-of-delivery uploads, integration logs, and shipment records can accumulate beside delivery-critical background work. A retention sweep that floods the same rate-limited workers may be correct about what to delete and still be wrong operationally: cleanup latency improves while shipment-event latency consumes its error budget. Don't accept that trade without naming it.

How should a daily cleanup job delete old uploads, logs, and records?

Start with a retention policy expressed as data, not a pile of timer callbacks inside Node.js or Express. Each data class needs an age threshold, a stable cursor, a maximum batch size, and an explicit order of operations. Database records that point to uploads normally cannot be treated like disposable log lines; references, legal holds, and concurrent readers change what “old” means. The application should decide eligibility from its own authoritative timestamps, while the scheduler should only decide when a sweep is allowed to begin.

For every candidate, use a state transition that can be repeated. Mark or select a bounded page, delete the associated object under the policy's rules, then remove or tombstone the record. If the process stops after either step, the next run must be able to continue without corrupting state. A cursor based on (created_at, id) is safer than an offset because rows can disappear while the sweep runs. Keep the cutoff fixed for the entire run; recalculating “now minus retention” on each page creates a moving boundary that is hard to audit.

There is another boundary people skip: cleanup does not deserve unlimited capacity merely because it runs at night. Logistics traffic follows time zones, carrier handoffs, and batch imports, so “off peak” is an assumption to verify. Give the sweep a concurrency ceiling and a stop condition tied to the foreground SLO. If the worker queue's oldest-message age or completion latency approaches its budget, pause cleanup dispatch. Slow is fine.

The useful invariant is: no retention work can consume the last unit of capacity reserved for shipment processing.

Queue acknowledgements matter once the sweep is split into batches. A consumer should acknowledge a cleanup batch only after its intended state transition is durable; an unacknowledged delivery can then be presented again, which is why the handler must tolerate repetition. RabbitMQ documents consumer acknowledgements separately from publisher confirms, and that distinction is important: confirmation that a broker accepted a message does not prove that a worker completed deletion. Google Cloud Pub/Sub likewise describes an asynchronous, scalable messaging service, but a product category does not remove the need to design idempotency, acknowledgement timing, and backlog alarms.

The incident lesson is capacity ownership

Consider a bounded production review, not a claimed benchmark: one scheduler opens a retention run while a rate-limited pool is already draining shipment jobs. The sweep discovers 48,000 eligible upload references and immediately publishes all of them. Ten workers share a downstream limit, cleanup messages sit beside delivery messages, and both job types retry transient throttling. Nothing in the cron expression is malformed. The architectural error is that discovery converted a low-priority maintenance task into an unbounded burst. The first symptom could be an older-message-age alarm on the delivery queue rather than a cleanup error. That is precisely why I wouldn't use “the cleanup completed” as the success criterion. The sweep has two SLO-facing outcomes: eligible data is removed within the retention window, and shipment work retains its latency budget while that happens. A dashboard needs the run ID, fixed cutoff, pages scanned, items selected, items completed, retries, oldest cleanup age, and delivery-queue age. It also needs a count of records skipped because of holds or failed preconditions; silently treating those as deleted makes the audit trail fiction. Teams sometimes reach for a timer in the web process because it is one dependency fewer on a diagram. That reasoning ends when Express runs more than one replica: every replica may fire, deployments may interrupt the callback, and ownership becomes implicit. This is an architectural failure mode, not a claim about a particular timer package. A database lease can elect one runner, but then the team owns lease expiry, clock assumptions, fencing, and observability. The “simple” timer has become a scheduler.

Stop there.

I am not sure what concurrency limit is right for an arbitrary logistics system, because the answer depends on the measured downstream quota, foreground arrival rate, deletion latency, and acceptable retention lag. A load test with production-shaped object sizes and a replay of peak shipment arrivals would resolve it. Until then, start with an explicitly small batch and reserve capacity rather than estimating from an empty queue.

The preventative shape is therefore two-level scheduling: one daily trigger creates a run with a fixed cutoff, then a dispatcher releases bounded pages only while the delivery SLO has headroom. Use separate queues or explicit priority only if their isolation semantics are understood; a priority label inside one saturated worker pool is not capacity reservation — it is a preference.

Put the preventative boundary in code

The following Go sketch keeps the scheduler generic. A daily trigger calls Start, the repository claims a stable page beneath one cutoff, and the dispatcher refuses to publish when foreground work has crossed its configured guardrail. There is no vendor endpoint to guess and no SDK hidden in the control path.

package retention

import (
    "context"
    "errors"
    "time"
)

type Candidate struct {
    ID        string
    CreatedAt time.Time
    Kind      string
}

type Page struct {
    Items      []Candidate
    NextCursor string
}

type Repository interface {
    CreateRun(ctx context.Context, runID string, cutoff time.Time) error
    ClaimPage(ctx context.Context, runID, cursor string, limit int) (Page, error)
}

type Queue interface {
    PublishCleanup(ctx context.Context, runID string, item Candidate) error
}

type ForegroundSignal interface {
    OldestShipmentAge(ctx context.Context) (time.Duration, error)
}

type Dispatcher struct {
    Repo             Repository
    Queue            Queue
    Foreground       ForegroundSignal
    BatchSize        int
    ShipmentAgeLimit time.Duration
}

var ErrForegroundBusy = errors.New("foreground capacity is reserved")

func (d Dispatcher) Start(ctx context.Context, runID string, cutoff time.Time) error {
    if err := d.Repo.CreateRun(ctx, runID, cutoff); err != nil {
        return err
    }

    cursor := ""
    for {
        age, err := d.Foreground.OldestShipmentAge(ctx)
        if err != nil {
            return err
        }
        if age >= d.ShipmentAgeLimit {
            return ErrForegroundBusy
        }

        page, err := d.Repo.ClaimPage(ctx, runID, cursor, d.BatchSize)
        if err != nil {
            return err
        }
        for _, item := range page.Items {
            if err := d.Queue.PublishCleanup(ctx, runID, item); err != nil {
                return err
            }
        }
        if page.NextCursor == "" {
            return nil
        }
        cursor = page.NextCursor
    }
}

CreateRun should enforce uniqueness for the run ID, while ClaimPage should preserve the fixed cutoff and stable ordering. The consumer needs its own idempotency key, commonly the policy version plus object ID, because publication and acknowledgement are separate events. A redelivery must converge on the same final state. An already absent upload can count as the desired deletion outcome only when the database transition and audit record agree; don't turn every missing object into success without checking why it is missing.

The process around the code matters more than the loop. Test a page containing an object under hold, an object concurrently referenced, an already completed item, and a batch redelivered before acknowledgement. Deploy the consumer with concurrency capped below the downstream quota, then enable the daily trigger. Alert on retention lag and foreground budget consumption, not on “cron ran.” For rollback, disable new run creation first and let claimed work reach a known state; killing workers while continuing to dispatch merely moves uncertainty into the queue.

Choose the ownership model, not the shortest setup

My default for this logistics case is an external daily trigger plus a durable, rate-limited cleanup queue, because the worker pool already has a backlog problem to manage and retention work must survive beyond one process lifetime. The catch is the extra operational surface. It is not suitable when the entire dataset is small enough for one bounded transaction, one host is genuinely stable, and a missed run is easy to detect and recover; stick with host cron there. Conversely, use a workflow engine rather than this pattern when cleanup has long-running, multi-step compensation or human approval, because a queue and a run table can become a poorly specified workflow system.

Cost is not just the scheduler's line item. Capacity planning should count database reads, object deletes, queue operations, retry amplification, observability retention, and on-call time. Latency has two clocks: time until stale data is removed and time imposed on shipment work. Write both budgets before comparing services. A managed trigger may cost more than cron while reducing ambiguous ownership; a self-managed queue may avoid a service bill while adding pager load. There is no honest winner without the team's traffic envelope and staffing model.

For the original Express application, keep the HTTP service out of execution. It may expose an authenticated trigger that creates a run, but it should return after durable acceptance rather than hold a request open until deletion finishes. The cleanup workers can be written in any language; the Go boundary above demonstrates the wire-independent contract. What matters is that scheduler, dispatcher, and consumer ownership are visible, testable, and separately observable.

References

• RabbitMQ consumer acknowledgements and publisher confirms: https://www.rabbitmq.com/docs/confirms
• Google Cloud Pub/Sub overview: https://cloud.google.com/pubsub/docs/overview

Further reading

• https://www.rabbitmq.com/docs/confirms
• https://cloud.google.com/pubsub/docs/overview