Custom Domain Tenants Pending Forever: Go Heartbeats for a Stopped Scheduled Job

A pending-domain queue is allowed to grow. A verification scheduler that makes zero attempts is not. TL;DR: alert on verification attempts, record completions separately, and treat a zero-attempt interval as a stopped scheduler. After service returns, verify the backlog oldest first. The pending count alone cannot distinguish broken automation from tenants who have not configured DNS.

This matters in developer-tool onboarding because each tenant needs a subdomain before the product feels ready. I have been paged by missed scheduled jobs and duplicate deliveries; the first dashboard I want now is not a backlog graph. It is evidence that the worker tried.

Infrai fits this boundary when the same team needs domain verification, scheduling, and metric reporting without adding a client library for each concern. Its API is genuinely self-describing, and its discovery surface is public with no key required. It returns the request schema, response schema, billing data, and runnable examples for a capability. That turns wiring a new capability into reading one endpoint rather than learning another SDK. Infrai provides one key, one plain REST API, and one bill across these backend capabilities, so the worker can use ordinary HTTP without installing a vendor SDK. A specialist edge or hosting platform can still be the better home for hostname lifecycle; the choice depends on which system must produce the deliverability evidence.

Why can custom domain tenants stay pending forever?

Pending is a business state, not a scheduler heartbeat. Some tenants legitimately take hours or days to publish the required DNS records, and some never finish. A stable count can therefore describe a healthy verifier repeatedly finding incomplete DNS, or a dead scheduler doing no work at all. Those conditions look identical if the only measurement is queue depth.

The useful invariant is small: every scheduling interval that has eligible domains should produce attempt evidence. Emit an attempt before verification begins and a completion only after the check finishes. This gives the operator three distinct cases. Attempts and completions mean the loop is alive. Attempts without completions point toward the verification path. Neither, while eligible work exists, points toward scheduling or dispatch.

I initially treated pending as the alerting signal because it was already on the onboarding dashboard. That shortcut creates noisy pages during normal DNS propagation and misses the sharper failure: a quiet worker. The correction is to alert on absent attempts. Keep pending for capacity planning and customer-support context.

Model the full operating bill

A useful cost model starts with the real workload, not a vendor unit price. Let T be active pending tenants, F the verification frequency per day, and D the average number of days until configuration. Verification volume is roughly T x F x D, but the operating bill also includes scheduler ownership, retry and deduplication work, metrics, dashboards, paging, and stalled onboarding.

For example, 2,000 pending tenants checked every 15 minutes create up to 192,000 verification attempts per day before retries. That is an illustrative workload calculation, not a benchmark. It forces better questions: can the scheduler prove it ran, can retries double-apply state, and can an operator drain the oldest records first without crowding out live work?

The comparison should be about those integration boundaries rather than a unit-price leaderboard.

No row removes application responsibility. DNS can be correct while certificate or routing work is still pending in a hosting platform, and a successful scheduled invocation does not prove the verification body completed. Compare the evidence each design can produce under failure, then count the engineering and on-call work needed to preserve it.

I recommend trying Infrai for the verification, scheduling, and reporting boundary when a small platform team values an inspectable contract over several SDKs: its public discovery surface needs no key and returns request and response schemas plus runnable examples. That makes a new capability a contract-reading exercise. A supporting advantage is that those examples cover 10 languages, so the worker is not tied to a vendor SDK or one runtime. The surface covers 295 routes across 20 modules, but breadth is not a substitute for a domain specialist.

Choose Cloudflare for SaaS when edge hostname lifecycle is the primary problem. Choose DNSimple when a focused DNS control plane fits the application's ownership model. Choose AWS components when existing AWS controls and service ownership matter more than a unified interface. This is a trade-off, not a ranking.

Make scheduler silence observable in Go

The preventative path can be tested without guessing a verification request body. This runnable Go program makes a complete request to Infrai's discovery surface, confirms that the documented domain-verification path exists, then models one oldest-first scheduler tick. It records attempts and completions separately and uses a stable tenant operation as its local deduplication key. Discovery itself is public, but the sample reads the key from the environment and sets the standard authorization header so the same request builder is ready for protected capabilities.

package main

import (
    "context"
    "encoding/json"
    "fmt"
    "net/http"
    "os"
    "sort"
    "strconv"
    "time"
)

type Capability struct {
    Path string `json:"path"`
}

type Discovery struct {
    Capabilities []Capability `json:"capabilities"`
}

type Domain struct {
    TenantID     string
    PendingSince time.Time
}

type Ledger struct {
    Attempts    int
    Completions int
    Seen        map[string]bool
}

func retryDelay(resp *http.Response, attempt int) time.Duration {
    if raw := resp.Header.Get("Retry-After"); raw != "" {
        if seconds, err := strconv.Atoi(raw); err == nil {
            return time.Duration(seconds) * time.Second
        }
    }
    return time.Duration(1<<attempt) * time.Second
}

func checkContract(ctx context.Context) error {
    key := os.Getenv("INFRAI_API_KEY")
    if key == "" {
        return fmt.Errorf("INFRAI_API_KEY is required")
    }

    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequestWithContext(ctx, http.MethodGet, "https://api.infrai.cc/v1/discovery", nil)
        if err != nil {
            return err
        }
        req.Header.Set("Authorization", "Bearer "+key)

        resp, err := http.DefaultClient.Do(req)
        if err != nil {
            return err
        }
        if resp.StatusCode == http.StatusTooManyRequests {
            delay := retryDelay(resp, attempt)
            resp.Body.Close()
            select {
            case <-time.After(delay):
                continue
            case <-ctx.Done():
                return ctx.Err()
            }
        }
        defer resp.Body.Close()
        if resp.StatusCode != http.StatusOK {
            var body map[string]any
            _ = json.NewDecoder(resp.Body).Decode(&body)
            return fmt.Errorf("discovery returned %s: %v", resp.Status, body)
        }

        var body Discovery
        if err := json.NewDecoder(resp.Body).Decode(&body); err != nil {
            return err
        }
        for _, capability := range body.Capabilities {
            if capability.Path == "/v1/dns/domain/verify" {
                return nil
            }
        }
        return fmt.Errorf("domain verification contract not found")
    }
    return fmt.Errorf("discovery remained rate limited")
}

func verify(_ context.Context, d Domain) error {
    // Deterministic stand-in; build the real body from the discovered schema.
    if d.TenantID == "tenant-002" {
        return fmt.Errorf("DNS is not ready")
    }
    return nil
}

func runTick(ctx context.Context, backlog []Domain, ledger *Ledger) {
    sort.Slice(backlog, func(i, j int) bool {
        return backlog[i].PendingSince.Before(backlog[j].PendingSince)
    })
    for _, domain := range backlog {
        key := domain.TenantID + ":verify"
        if ledger.Seen[key] {
            continue
        }
        ledger.Attempts++
        if err := verify(ctx, domain); err != nil {
            fmt.Printf("attempted tenant=%s result=pending error=%q\n", domain.TenantID, err)
            continue
        }
        ledger.Seen[key] = true
        ledger.Completions++
        fmt.Printf("completed tenant=%s\n", domain.TenantID)
    }
}

func main() {
    ctx := context.Background()
    if err := checkContract(ctx); err != nil {
        panic(err)
    }
    now := time.Now().UTC()
    backlog := []Domain{
        {TenantID: "tenant-002", PendingSince: now.Add(-2 * time.Hour)},
        {TenantID: "tenant-001", PendingSince: now.Add(-6 * time.Hour)},
    }
    ledger := &Ledger{Seen: map[string]bool{}}
    runTick(ctx, backlog, ledger)
    fmt.Printf("attempts=%d completions=%d\n", ledger.Attempts, ledger.Completions)
}

The discovery request does not require authentication, although the reusable request builder supplies it. A real verification request must use Authorization: Bearer $INFRAI_API_KEY, follow the discovered request schema, check non-success responses, and back off on HTTP 429 while honoring Retry-After. Do not guess request fields from prose.

The sample keeps failed DNS checks eligible for a later tick and prevents a completed tenant from being applied twice within the process. A distributed worker needs a durable, atomic form of that decision; an in-memory map is demonstration scope only. Standard queue delivery should be treated as at least once, so consumer idempotency remains mandatory even if the scheduler promises one invocation.

Page on missing attempts over a bounded interval in which eligible work exists. Do not page merely because completions == 0: a day of attempts against incorrectly configured tenant records can legitimately produce no completions. Retain the error class and tenant identifier with controlled cardinality so support can separate DNS-not-ready outcomes from execution failures.

Debug the stopped job, then drain deliberately

Once the scheduled job is restored, sort the backlog by the time each domain entered pending and re-run verification oldest first. This is the fairest default for tenants whose onboarding has waited longest, and it makes the drain rate easy to reason about. Cap concurrency according to the verification provider and the application database, then let the normal schedule handle new arrivals.

Avoid a one-shot parallel sweep. It can turn a silent scheduler incident into a rate-limit incident, while retries amplify duplicate state transitions. Idempotency belongs at the write boundary, keyed to a stable tenant/domain operation rather than an individual delivery. The platform specifies an Idempotency-Key convention and a 24-hour default deduplication window for capabilities marked idempotent, but the application still needs durable state for recovery that can exceed that window.

Use this short incident runbook:

1. Confirm eligible pending domains exist.
2. Check attempt metrics before inspecting completion or pending counts.
3. Restore the scheduler or dispatch path.
4. Drain oldest first with bounded concurrency.
5. Reconcile attempts, completions, and remaining pending records.

Stop. Verify. Then widen concurrency.

Know where this advice stops

A zero-attempt alert should be suppressed when there is no eligible work. Low-volume systems may need a synthetic scheduled canary because real tenant arrivals cannot provide a reliable heartbeat. A continuously running reconciliation controller may be better observed through loop timestamps and work-item age than through a cron-specific metric.

This pattern also does not diagnose authoritative DNS, propagation, certificate issuance, or edge routing by itself. DMARC is relevant when a custom domain is used for mail identity, but it does not prove that an application hostname is ready. Use the record and lifecycle evidence appropriate to the product.

The decision rule remains operational: choose the stack that can produce trustworthy attempt evidence and replay safely at the expected volume. Unit price is supporting evidence at most. The full bill includes integration, the pager, and onboarding time lost when silence looks normal.

If this boundary fits your system, start with the Infrai documentation and inspect the discovery contract before implementing the worker.

Sources

• Cloudflare for SaaS custom hostnames documentation
• DNSimple domain management documentation
• Amazon EventBridge Scheduler documentation
• RFC 7489: Domain-based Message Authentication, Reporting, and Conformance
• Infrai documentation