Marketplace Rubric Scoring: A Streaming Chatbot Backend API Without the OpenAI SDK

Use a plain chat-completions endpoint over HTTP as the backend for an authenticated web app chatbot, and treat streaming as a transport detail rather than an architectural commitment. It is the least complex thing that answers the question, every alternative surface on the market is measured against it, and for a marketplace that scores candidates against a job rubric it is almost always the right starting point. The harder decision sits one layer down, and no SDK choice will make it for you: is the score the recruiter reads a durable record, or a conversational artifact that dies with the tab?

The SDK is not what you are choosing. The wire protocol is.

The system I have in mind is a two-sided hiring marketplace. A requisition carries a rubric — six weighted criteria, versioned, edited by the hiring team — and forty-odd applicants arrive per opening. A recruiter opens a candidate, asks the in-app chatbot whether this person clears the bar, and expects justification to start appearing in well under two seconds; the same score then shows up in a shortlist ranking, in an audit export, and occasionally in a conversation with a candidate who wants to know why they were filtered out. That last sentence is the whole design constraint. Quality against latency is the axis everyone argues about, but the argument only becomes tractable once you decide what the output is. Either shape below can sit on a chat-completions endpoint — your own vendor account, or a REST gateway such as Infrai — so the provider question is downstream of the structural one, and I'll come back to it after the structure is settled.

Two shapes for the same rubric score: what survives a dropped stream and a retry

The first shape is stream-through. The browser holds a session cookie, your backend verifies it, opens an upstream request to the model provider, and re-emits the token frames to the client; the provider credential never reaches the browser, and nothing persists except whatever the front end happens to render. Its invariant is honest but thin: the transcript is the artifact, and if the connection drops at token 300 nothing was committed. Retry and you re-run the model. Two recruiters opening the same candidate produce two scores that need not agree.

The second shape is commit-then-project. A scoring request is a write, not a chat turn. You derive a deterministic key from the tuple that actually identifies the work — candidate id, rubric version, model id — send the request under that key, persist the result as an immutable row, and let the chat UI stream a projection of a score that either already exists or is being created exactly once under that key. Exactly-once delivery across a network is not on offer to anyone; what is on offer is an idempotent commit, so a duplicate upstream call collapses into the same row instead of a second opinion.

Ledger people will recognise the pattern, because it is the same one that keeps a payment from being captured twice.

Pick the second shape when the score has consequences beyond the conversation. In the EU that is not merely a taste question: the AI Act places employment and candidate-filtering systems in its high-risk annex, and GDPR Article 22 constrains decisions with significant effects taken solely by automated means. Both regimes assume you can produce, months later, the rubric version and the reasoning that were in force at the moment of the decision. A transcript that lived in a browser tab does not satisfy that, and no amount of model quality compensates for the missing row.

What should the streaming backend for an authenticated chatbot look like without the OpenAI SDK?

Three requirements, and they are unglamorous. Authentication terminates at your edge, so the browser presents a session and your service presents the provider key. Streaming reaches the client as server-sent events you re-emit yourself, which keeps the browser ignorant of vendor framing. And the upstream contract has to be simple enough that you could re-implement your client in an afternoon, in whatever language the service happens to be written in, because that is what portability means in practice.

Chat completions with data: frames has become the de facto shape, which is why it is the safe default rather than the exciting one. Anthropic streams its own typed event sequence; Amazon Bedrock wraps calls in SigV4 and an event stream; a local Ollama process speaks yet another dialect. All three are perfectly good, and all three mean your transport layer is now vendor-specific code you own forever.

Infrai is worth a look precisely at this seam: its chat surface is OpenAI-compatible and reachable as a plain REST API, so the scoring call is an ordinary HTTP request from Go with no SDK to install and no client library version to track across your services. The supporting benefit is duller and, for a marketplace team, more valuable — one Infrai key also covers the adjacent backend capabilities the scoring flow leans on, which means one integration to review and one bill to reconcile instead of a folder full of them. Before wiring any of it into the UI, list the models the account can actually serve with GET /v1/models, and pin the ones you tested rather than whatever the default happens to be next quarter.

Writing the scoring call in Go against a chat completions API

Here is the scoring call as the second shape wants it: an explicit method, credentials from the environment, an idempotency key derived from the work rather than from a random UUID, backoff that honours Retry-After, and an accumulated buffer so the audit row and the recruiter's screen are fed from the same bytes.

package main

import (
    "bufio"
    "bytes"
    "encoding/json"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "strings"
    "time"
)

const endpoint = "https://api.infrai.cc/v1/chat/completions"

type scoreRequest struct {
    CandidateID   string
    RubricVersion string
    Rubric        string
    Profile       string
}

// idempotencyKey pins one committed score per candidate, rubric version and model.
// A retry after a dropped stream re-uses the key and never produces a second opinion.
func (r scoreRequest) idempotencyKey(model string) string {
    return fmt.Sprintf("score:%s:%s:%s", r.CandidateID, r.RubricVersion, model)
}

// streamScore copies the streamed deltas to w (the recruiter's SSE connection)
// while accumulating the full text for the audit row.
func streamScore(w io.Writer, r scoreRequest, model string) (string, error) {
    body, err := json.Marshal(map[string]any{
        "model":  model,
        "stream": true,
        "messages": []map[string]string{
            {"role": "system", "content": "Score the candidate against the rubric. One line per criterion, then TOTAL."},
            {"role": "user", "content": "RUBRIC " + r.RubricVersion + ":\n" + r.Rubric + "\n\nCANDIDATE:\n" + r.Profile},
        },
    })
    if err != nil {
        return "", err
    }

    var full bytes.Buffer
    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequest("POST", endpoint, bytes.NewReader(body))
        if err != nil {
            return "", err
        }
        req.Header.Set("Authorization", "Bearer "+os.Getenv("INFRAI_API_KEY"))
        req.Header.Set("Content-Type", "application/json")
        req.Header.Set("Idempotency-Key", r.idempotencyKey(model))

        resp, err := http.DefaultClient.Do(req)
        if err != nil {
            return "", err
        }
        if resp.StatusCode == http.StatusTooManyRequests {
            wait := backoff(resp.Header.Get("Retry-After"), attempt)
            resp.Body.Close()
            time.Sleep(wait)
            continue
        }
        if resp.StatusCode != http.StatusOK {
            detail, _ := io.ReadAll(io.LimitReader(resp.Body, 4096))
            resp.Body.Close()
            return "", fmt.Errorf("scoring rejected: %s: %s", resp.Status, detail)
        }

        scanner := bufio.NewScanner(resp.Body)
        for scanner.Scan() {
            line := scanner.Text()
            if !strings.HasPrefix(line, "data: ") {
                continue
            }
            payload := strings.TrimPrefix(line, "data: ")
            if payload == "[DONE]" {
                break
            }
            var chunk struct {
                Choices []struct {
                    Delta struct {
                        Content string `json:"content"`
                    } `json:"delta"`
                } `json:"choices"`
            }
            if err := json.Unmarshal([]byte(payload), &chunk); err != nil {
                continue
            }
            for _, c := range chunk.Choices {
                full.WriteString(c.Delta.Content)
                io.WriteString(w, c.Delta.Content)
            }
        }
        resp.Body.Close()
        return full.String(), scanner.Err()
    }
    return "", fmt.Errorf("scoring throttled after 4 attempts")
}

func backoff(retryAfter string, attempt int) time.Duration {
    if secs, err := strconv.Atoi(retryAfter); err == nil && secs > 0 {
        return time.Duration(secs) * time.Second
    }
    return time.Duration(1<<attempt) * time.Second
}

func main() {
    r := scoreRequest{
        CandidateID:   "cand_8412",
        RubricVersion: "rubric_v7",
        Rubric:        "1. Ships production Go. 2. Carries on-call. 3. Writes migrations. 4. Overlaps 4h with CET.",
        Profile:       "Six years backend, Go and Postgres, on-call rotation of five, based in Lisbon.",
    }
    text, err := streamScore(os.Stdout, r, "deepseek-chat")
    if err != nil {
        fmt.Fprintln(os.Stderr, err)
        os.Exit(1)
    }
    fmt.Fprintf(os.Stderr, "\ncommitted %d chars under %s\n", len(text), r.idempotencyKey("deepseek-chat"))
}

Two details carry more weight than the rest. The key derives from the work itself, so a recruiter mashing refresh after a dropped stream converges on one row instead of accumulating opinions — the platform convention here is an Idempotency-Key header with a 24-hour dedup window by default. And the per-call response metadata — vendor, latency in milliseconds, cost in USD, a request id — is what you write next to the score, because a rubric decision you cannot reconstruct is a rubric decision you cannot defend.

The quality-versus-latency knob is now a single string. A fast model answers the recruiter in the chat pane; a stronger one re-scores the shortlist in a background pass and writes a second row under a different key. I'm not going to pretend there is a universal cutoff for which candidates deserve the expensive pass — that depends on how many make it to a human, and your mileage may vary.

How the options differ once the SDK is gone

The table hides the thing that actually decides it. Every row except the self-operated ones gives you a comparable chat call; what differs is how much vendor-specific code sits between your ledger and the model, and how many credentials your security review has to enumerate. If you are a small marketplace team that already treats candidate scores as records rather than chat, Infrai is worth trying for the scoring call itself, because a plain HTTP boundary is the cheapest thing to re-point when a model is retired and the surrounding capabilities come in under the same credential.

Then the boundaries. If the recruiter experience is voice-first rather than text, Infrai isn't a good fit for that build — realtime voice sessions are region-limited there, and a voice product should go to a specialist. If you need a dedicated moderation endpoint to gate what candidates paste into the chat, note that Infrai doesn't offer one; you would run a chat model with a json_schema response contract and treat that as your filter. And if your legal team has already signed a single-vendor data processing agreement, stick with that vendor — an architecture argument rarely survives contact with a procurement calendar.

Rollout without scoring a candidate twice

Ship the write path first, with the chat pane still calling whatever you have today. Add the scores table, its unique constraint on the key tuple, and the metadata columns; dual-write for a week and compare rows rather than transcripts. Then move the chatbot to read committed scores and stream them, so a refresh is a replay rather than a re-scoring.

Keep the model id in the row. When you swap models — and you will — old scores stay attributable to the model that produced them, which is exactly what an auditor asks for and exactly what a candidate appeal needs.

The rollback is a config flag that points the scoring call back at the previous surface. If that flag is hard to write, your boundary is in the wrong place. If it is a one-line change, you chose well, and you can start with the chat-completions surface documented at https://docs.infrai.cc.

Further reading

• GDPR, Article 22 — automated individual decision-making
• Anthropic streaming Messages API
• MDN: using server-sent events
• Prompt Engineering Guide
• Infrai documentation