Batch Enqueue Jobs: 5 Rules for Rate-Limited Background Email and Import Workers

Short answer: batch publishing is the right way to enqueue a large customer-support cleanup, email send, import, or backfill without holding a web request open, provided each work item remains a separate message and every worker is both rate-limited and idempotent. The batch improves admission efficiency; it does not make downstream capacity unlimited.

That distinction decides the architecture. A request handler should validate the operation, create stable job IDs, publish the batch, and return. Workers can then make progress independently, acknowledge successful items, and retry individual failures without replaying the entire customer action. Fast admission and controlled execution are different SLOs, and I would budget them separately.

What governance should batch enqueue background jobs enforce for a rate-limited worker queue?

Consider a bounded production scenario rather than an invented benchmark. A support administrator starts a periodic cleanup of 10,000 stale conversation records; each cleaned record may also enqueue an email or an account-import update. The web tier publishes quickly and returns, so request latency looks healthy. Thirty-two workers then wake at once and call a downstream API that responds with HTTP 429 when its quota is exhausted. The queue did its job. The system still missed its processing objective because concurrency, retry timing, and downstream capacity were never part of the admission decision.

I use this as a capacity-planning exercise: arrival rate, sustainable service rate, and backlog age belong on the same page. If jobs arrive at 100 per second while the dependency safely accepts 40 per second, no batch size repairs the 60-per-second deficit. It only changes how quickly the deficit becomes visible. Set a worker-side rate limit below the known downstream ceiling, add exponential backoff that honors Retry-After, and alert on oldest-message age rather than celebrating a low publish latency.

Keep the unit of recovery small.

One message per conversation, email, or imported account lets workers acknowledge successes independently. Putting 10,000 records into one message couples their fate, pushes payload limits, and makes an ambiguous retry expensive. A practical service objective might distinguish “accepted within the request latency budget” from “99% processed before the cleanup deadline”; the exact numbers depend on observed traffic and the downstream contract. I'm not sure what limit is safe for your dependency until its quota and burst behavior are measured, and your mileage may vary across tenants.

Use five rules.

1. Assign one stable ID to each work item before publishing. That ID is the consumer's idempotency key, not a random value regenerated on every retry.
2. Publish many messages in one batch, but keep each message independently consumable and acknowledgeable.
3. Throttle where the external call happens. A queue without native debounce or throttle cannot infer the quota of an email, CRM, or import API.
4. Treat standard delivery as at-least-once. Record completion by stable ID before acknowledging the message.
5. If several distinct consumers need the same logical event, publish explicitly to one queue per consumer; do not assume a single-topic, multi-subscriber fan-out primitive.

The fourth rule is the one teams skip under schedule pressure. An acknowledgment can be lost after the external side effect succeeds, so the same message can arrive again. The consumer must recognize that job ID and avoid sending the same email or applying the same cleanup twice. Exactly-once language in a diagram doesn't remove that boundary — the receiver owns it.

Migration paths across managed queues, libraries, and workflow engines

The queue choice should follow the recovery model and the team's willingness to carry on-call load. Brand familiarity is weak evidence; an explicit buy-versus-build review is better.

This is not a universal recommendation. Stick with an existing cloud queue when its operational model is already understood and moving would add abstraction without reducing toil. Choose BullMQ, Celery, or Sidekiq when the application stack fits and owning the broker is an accepted part of the on-call budget. Choose Temporal or Airflow when the real requirement is a workflow graph; a simple queue has no DAG or fan-out/join primitive, and pretending otherwise moves orchestration into application code where it is harder to inspect.

The plain HTTP option is strongest when language neutrality matters — any worker that can make an authenticated request can use it — and when a platform team wants one consistent integration surface rather than another SDK lifecycle. The boundary matters as much as the convenience: delayed messages stop at seven days, message bodies at 256 KB, retention at 30 days, and acknowledged messages are deleted rather than retained for Kafka-style replay or multiple consumer groups. FIFO deduplication covers only a five-minute window. Those constraints are suitable for bounded background work, not an event archive.

Integration design: a replaceable Go publishing adapter

This runnable publisher calls Infrai's verified batch route with an explicit method, bearer authentication, an idempotency key, status checks, and bounded retry on HTTP 429. The exact batch request schema is intentionally not duplicated here: retrieve it from the public queue.publish discovery document, validate the one-message-per-item payload during deployment, and pass that JSON in INFRAI_BATCH_JSON. Set INFRAI_BASE_URL to the documented API base. This keeps a copied article from freezing request fields while still making the publishing and failure behavior concrete.

package main

import (
    "bytes"
    "context"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "strings"
    "time"
)

const publishPath = "/v1/queue/publish_batch"

func retryDelay(header string, attempt int) time.Duration {
    if seconds, err := strconv.Atoi(header); err == nil && seconds >= 0 {
        return time.Duration(seconds) * time.Second
    }
    return time.Duration(1<<attempt) * 500 * time.Millisecond
}

func publishBatch(ctx context.Context, client *http.Client, baseURL, key, idempotencyKey string, body []byte) error {
    for attempt := 0; attempt < 5; attempt++ {
        req, err := http.NewRequestWithContext(ctx, http.MethodPost, strings.TrimRight(baseURL, "/")+publishPath, bytes.NewReader(body))
        if err != nil {
            return err
        }
        req.Header.Set("Authorization", "Bearer "+key)
        req.Header.Set("Content-Type", "application/json")
        req.Header.Set("Idempotency-Key", idempotencyKey)

        resp, err := client.Do(req)
        if err != nil {
            return fmt.Errorf("publish batch: %w", err)
        }
        responseBody, readErr := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
        resp.Body.Close()
        if readErr != nil {
            return fmt.Errorf("read response: %w", readErr)
        }
        if resp.StatusCode >= 200 && resp.StatusCode < 300 {
            fmt.Println(string(responseBody))
            return nil
        }
        if resp.StatusCode != http.StatusTooManyRequests {
            return fmt.Errorf("publish batch: status %d: %s", resp.StatusCode, strings.TrimSpace(string(responseBody)))
        }

        timer := time.NewTimer(retryDelay(resp.Header.Get("Retry-After"), attempt))
        select {
        case <-timer.C:
        case <-ctx.Done():
            timer.Stop()
            return ctx.Err()
        }
    }
    return fmt.Errorf("publish batch: retry limit reached")
}

func main() {
    baseURL := os.Getenv("INFRAI_BASE_URL")
    key := os.Getenv("INFRAI_API_KEY")
    body := []byte(os.Getenv("INFRAI_BATCH_JSON"))
    if baseURL == "" || key == "" || len(body) == 0 {
        panic("INFRAI_BASE_URL, INFRAI_API_KEY, and INFRAI_BATCH_JSON are required")
    }

    ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
    defer cancel()
    client := &http.Client{Timeout: 15 * time.Second}
    if err := publishBatch(ctx, client, baseURL, key, "support-cleanup-20260813", body); err != nil {
        panic(err)
    }
}

The stable request idempotency key protects retries of this publishing operation. Each message in INFRAI_BATCH_JSON still needs its own stable work ID so the consumer can suppress a duplicate delivery after an external side effect. Don't tight-loop on 429. The same rule belongs in the worker when it calls a rate-limited email or import API.

For a periodic cleanup, the scheduler should trigger enqueueing and finish, while workers perform the long operation. A cron execution can run for at most 900 seconds, so a cleanup that may exceed that window belongs in the queue. Paused schedules do not replay missed triggers, and trigger timing can have seconds of jitter, which means the job design should tolerate a late or skipped tick without manufacturing duplicate work.

Reliability after admission depends on service rate

Use batch enqueue when one customer action creates many independent items and request latency matters. Set batch size from payload and admission constraints, then size worker concurrency from measured downstream capacity. Track publish errors, oldest-message age, processing latency, retry count, dead-letter volume, and duplicate suppression; without those signals, the on-call engineer sees the backlog only after customers do.

Do not use this pattern for a task that must finish inside the original request, a payload above 256 KB, a delay beyond seven days, or a replayable event log. It is also not suitable when one event must automatically reach many independent subscribers, or when several steps require joins and durable workflow state. Pick a pub/sub product after verifying its delivery contract for the first case, Kafka-style storage for replay, and Temporal or Airflow for orchestration.

There is one final invariant: publishing a batch is an admission optimization, while correctness remains per message. Keep stable IDs, throttle at the dependency, acknowledge only completed work, and plan capacity from service rate rather than batch size. That's the difference between a fast endpoint and a background system that can meet an SLO.

References

• https://vercel.com/docs/cron-jobs
• https://en.wikipedia.org/wiki/Exponential_backoff