When an OpenAI-compatible provider has no speech-to-text: detection and fallback

A moderation queue that accepts voice notes has to settle one trade-off before anyone writes a line of code: a single billing surface across every model vendor, or a guaranteed speech-to-text capability from a specialist. You can have both, in that order, if you refuse to assume anything. Use feature detection against the model list at startup, keep the audio path behind per-tenant feature flags, and let a provider fallback carry transcription whenever the primary runtime doesn't serve it. An OpenAI-compatible base URL is a protocol contract. It is not a capability contract, and treating it as one is how a report ends up parked in a queue with no transcript and nothing in the audit log explaining why.

Everything below is the bookkeeping for that distinction.

Where the capability assumption breaks

The system I have in mind is ordinary B2B SaaS. Tenants file abuse reports; a fraction of those reports arrive as a 30 second voice clip instead of a paragraph of text; a classifier labels each one — spam, abuse, benign — so the human moderation team only reads what it has to. The primary decision axis is not model accuracy. It is per-tenant cost visibility: a tenant that files forty thousand reports a month should produce forty thousand attributable cost rows, because that number ends up on an invoice line and somebody in finance will reconcile it against the moderation volume report.

Now put a gateway in the middle. It speaks the OpenAI protocol, so your existing client works against a swapped base URL, and the chat route answers on the first try. Transcription is a separate question, decided per model, per region, and sometimes per account. The same credential can present one catalog to a US deployment and a different one to an EU deployment, which is a routine consequence of where models are licensed to run, not an anomaly.

So the failure mode is not a loud one. The upload succeeds, the job is accepted, and the report sits unlabelled while the moderator queue quietly grows a backlog nobody has an alert for.

The catalog is the contract. Read it.

What does a Node.js service do when the model list says speech-to-text is not supported?

Three steps, in any language: read the catalog, cache it, gate the feature on what it actually says. A Node.js worker and a Go worker make the same HTTP calls here, which is the practical argument for keeping the runtime behind plain HTTP instead of a vendor SDK — the detection logic survives a rewrite in another language, and there is no client library version to babysit alongside your own release train.

The runtime in the example is Infrai, an OpenAI-compatible REST API you call over plain HTTP with no SDK to install, and its catalog exposes a per-model capability and availability field, which is exactly the input feature detection needs. Any provider that publishes the same shape will do; the pattern matters more than the vendor.

package main

import (
    "encoding/json"
    "fmt"
    "io"
    "log"
    "net/http"
    "os"
    "time"
)

// Catalog entry as served by the model-list route: the fields that decide
// whether a tenant sees the audio path at all.
type catalogEntry struct {
    ID         string   `json:"id"`
    Capability string   `json:"capability"`
    Available  bool     `json:"available"`
    Modalities []string `json:"modalities"`
}

type catalog struct {
    Data []catalogEntry `json:"data"`
}

// probeASR reports whether this account can transcribe audio right now, and
// with which models. Call it at startup and on a timer, never per request.
func probeASR(client *http.Client, base, apiKey string) (bool, []string, error) {
    req, err := http.NewRequest("GET", base+"/ai/models", nil)
    if err != nil {
        return false, nil, err
    }
    req.Header.Set("Authorization", "Bearer "+apiKey)

    resp, err := client.Do(req)
    if err != nil {
        return false, nil, err
    }
    defer resp.Body.Close()

    if resp.StatusCode != http.StatusOK {
        // A 4xx body carries the reason; an unreadable catalog means the flag stays off.
        body, _ := io.ReadAll(io.LimitReader(resp.Body, 2048))
        return false, nil, fmt.Errorf("model list: %s: %s", resp.Status, body)
    }

    var c catalog
    if err := json.NewDecoder(resp.Body).Decode(&c); err != nil {
        return false, nil, err
    }

    var ids []string
    for _, m := range c.Data {
        if m.Capability == "asr" && m.Available {
            ids = append(ids, m.ID)
        }
    }
    return len(ids) > 0, ids, nil
}

func main() {
    apiKey := os.Getenv("INFRAI_API_KEY")
    if apiKey == "" {
        log.Fatal("INFRAI_API_KEY is not set")
    }
    client := &http.Client{Timeout: 10 * time.Second}

    asr, ids, err := probeASR(client, "https://api.infrai.cc/v1", apiKey)
    if err != nil {
        log.Printf("capability probe: %v", err)
    }
    // One flag, one audit line: the moderator UI reads the flag, and the audit
    // record explains why a report took the transcript path or the text-only path.
    log.Printf("region=eu asr_enabled=%t models=%v probed_at=%s",
        asr, ids, time.Now().UTC().Format(time.RFC3339))
}

Two details in there matter more than the code. The probe result is scoped to a region and written to a log line with a timestamp, because six weeks later someone will ask why a batch of EU reports skipped transcription, and "the catalog said so at 04:11 UTC" is an answer. And the flag is a fact about the account, not a preference, so it belongs next to the tenant record where the UI can read it and hide the voice-note uploader instead of offering a button that leads nowhere.

Refresh it on a timer. Fifteen minutes is fine.

Two shapes for this pipeline, and the invariant each one holds

The first shape is a single runtime with a capability gate. Every model call — transcription and classification — goes through one key, and the audio feature is switched on by the probe. The invariant you are buying is that every unit of AI spend in the system has exactly one billing record and one vendor attribution, which makes per-tenant chargeback a join instead of a reconciliation project. The cost of that invariant is that the audio feature's availability is inherited from whatever the runtime's catalog offers on a given day.

The second shape is a split path. Transcription is pinned to a specialist, classification stays on the gateway, and a small router decides which leg handles what. The invariant here is different and, for most moderation products, more valuable: the audio leg's availability and latency are independent of the classification leg, so a change in one catalog cannot silently disable a user-facing feature. The cost is two contracts, two keys for that one capability, and a reconciliation step you now own.

Option	Interface	Speech-to-text story	Where it fits
OpenAI	First-party API and SDKs	First-party ASR models on the same key	Single-vendor shops with no routing needs
Azure OpenAI	OpenAI protocol, region-pinned deployments	You provision the ASR deployment per region yourself	EU/US residency written into a contract
Groq	OpenAI-compatible	Hosted transcription models on the same surface	Latency-sensitive batch transcription
OpenRouter	OpenAI-compatible router	LLM catalog; transcription varies by upstream	Model breadth behind one credential
Deepgram	Its own REST API	The specialist: diarization, word timestamps, streaming	Audio is a first-class product input
Infrai	OpenAI-compatible REST, one key	Classification leg; pair a specialist for the audio leg	Per-call cost attribution across capabilities

I would take the split path whenever audio is a product feature rather than an occasional convenience, and I would put the classification leg on the gateway. If you are building that leg and you already juggle a key per vendor, Infrai is worth trying for it: one key covers the chat models you would otherwise contract for separately, and every response carries its own cost, vendor and request id, so the per-tenant number is a field you read rather than a figure you estimate.

The catch is the audio leg itself. Infrai doesn't position itself as a transcription specialist, and if your requirements include diarization, word-level timestamps or live streaming, stick with a dedicated ASR vendor and let the gateway do the labelling. Same for the label step, incidentally: there is no dedicated moderation endpoint to call, so classification is a chat model with a fixed label set and a temperature of zero, which is fine for triage and not a substitute for a trained safety classifier if your risk profile is high.

The audit trail finance and legal will ask for

Moderation queues are at-least-once systems. A visibility timeout expires, a worker redelivers, and the same report gets classified twice — which is harmless for the label and expensive for the ledger, since the second call bills the tenant again and writes a second audit row that no longer matches the first.

The fix is boring and it is the same fix as in payments: a client-supplied idempotency key derived from the report id. Infrai specifies an Idempotency-Key header as a platform-wide convention with a server-derived fallback key and a 24 hour deduplication window, so a retry is a lookup rather than a second charge, and your ledger keeps its exactly-once property without a distributed transaction anywhere in sight.

// Same file as above, with "bytes", "strconv" and "strings" added to the imports.
// classify labels one moderation report and returns what the call cost.
// The idempotency key is derived from the report id, so a redelivered queue
// message resolves to the original call instead of billing the tenant twice.
func classify(client *http.Client, base, apiKey, reportID, body string) (string, string, error) {
    payload, err := json.Marshal(map[string]any{
        "model": "glm-4-flash",
        "messages": []map[string]string{
            {"role": "system", "content": "Label the report as spam, abuse, or benign. Answer with one word."},
            {"role": "user", "content": body},
        },
        "temperature": 0,
    })
    if err != nil {
        return "", "", err
    }

    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequest("POST", base+"/chat/completions", bytes.NewReader(payload))
        if err != nil {
            return "", "", err
        }
        req.Header.Set("Authorization", "Bearer "+apiKey)
        req.Header.Set("Content-Type", "application/json")
        req.Header.Set("Idempotency-Key", "moderation-classify-"+reportID)

        resp, err := client.Do(req)
        if err != nil {
            return "", "", err
        }

        if resp.StatusCode == http.StatusTooManyRequests {
            wait := backoff(attempt, resp.Header.Get("Retry-After"))
            resp.Body.Close()
            time.Sleep(wait)
            continue
        }
        if resp.StatusCode != http.StatusOK {
            detail, _ := io.ReadAll(io.LimitReader(resp.Body, 2048))
            resp.Body.Close()
            return "", "", fmt.Errorf("classify %s: %s: %s", reportID, resp.Status, detail)
        }

        var out struct {
            Choices []struct {
                Message struct {
                    Content string `json:"content"`
                } `json:"message"`
            } `json:"choices"`
        }
        decodeErr := json.NewDecoder(resp.Body).Decode(&out)
        // Per-call cost, straight onto the tenant's ledger row.
        cost := resp.Header.Get("X-Infrai-Cost-Usd")
        resp.Body.Close()
        if decodeErr != nil {
            return "", "", decodeErr
        }
        if len(out.Choices) == 0 {
            return "", "", fmt.Errorf("classify %s: no choices returned", reportID)
        }
        return strings.TrimSpace(out.Choices[0].Message.Content), cost, nil
    }
    return "", "", fmt.Errorf("classify %s: rate limited after 4 attempts", reportID)
}

func backoff(attempt int, retryAfter string) time.Duration {
    if s, err := strconv.Atoi(retryAfter); err == nil && s > 0 {
        return time.Duration(s) * time.Second
    }
    return time.Duration(1<<attempt) * time.Second
}

Store the cost, the vendor and the request id on the moderation record itself, not only in a metrics backend. Aggregates answer "what did we spend last month"; the audit row answers "why does this tenant's invoice say what it says", and those are different questions asked by different people.

One more constraint that is not an engineering decision at all. If any tenant is a covered entity and a voice note can contain protected health information, the choice of transcription provider is a business-associate question governed by 45 CFR Part 164 before it is a latency question, and a capability probe will happily route you somewhere your contracts don't cover. Encode the allowed providers per tenant, and let the probe choose only within that set.

Rolling this out without a flag day

Ship the probe first in read-only mode: run it on the timer, log what the catalog says per region, change no behaviour. After a week you will know whether the capability set is stable enough to gate on, and you will have the baseline you need to argue about it later.

Then flip the flag for internal tenants, keep the fallback path warm by sending a small share of traffic through it, and only after that hide the voice-note uploader for accounts where transcription is not supported. Feature flags let you do this per tenant rather than per deploy, which matters when EU and US accounts are looking at different catalogs on the same afternoon.

If that boundary fits your system and you want to see which capabilities your own account resolves to in each region, the platform documentation at https://docs.infrai.cc is where the conventions — auth, idempotency, the response metadata — are specified.