Support Review Telemetry — Compare Small Models via Batch Processing and Token Counting

Short answer: reduce LLM cost by routing support-code reviews by observed difficulty, letting a small model summarize, classify, and extract JSON, then escalating only when schema, evidence, or latency signals cross a threshold.

The deciding constraint is quality versus latency, not the name on a model card. A cheap response that invents a file location is expensive. A perfect review returned after the support engineer has already merged the patch is useless. Measure both outcomes per request, then compare candidates against the same acceptance policy and current rate cards.

This makes the cost question operational. Prompt token counting estimates exposure before a call. Recorded input and output usage explain it afterward. JSON validation catches malformed findings immediately, while sampled human review catches the harder failure: valid JSON containing weak engineering judgment.

Change the mental model from one call to an observable review lane

The before picture is familiar: a code diff enters one large-model prompt, a JSON blob comes back, and a dashboard shows a monthly total. Summarization, classification, extraction, and review reasoning are tangled together. When spend rises, nobody can tell whether the cause was larger diffs, verbose prompts, retries, or more output. When quality falls, the aggregate cost chart offers no clue.

The after picture is a lane with checkpoints. In words: diff arrives; a local gate records its size and requested task; a candidate produces structured findings; deterministic code validates the envelope and evidence locations; the request either exits or escalates; then an asynchronous sample reaches a human reviewer. Every arrow emits one compact event.

That last sentence matters.

For a customer-support team reviewing code changes, the useful unit is not “one LLM call.” It is one accepted review. Give that review a stable ID and record the candidate class, task, prompt tokens, output tokens, queue time, generation time, validation result, escalation reason, and final disposition. Keep source code and prompt contents out of routine telemetry unless the team has explicitly designed the retention and access controls; identifiers and counts are enough for the routing analysis described here.

The crisp before/after is this: before, cost per call with unknown quality; after, cost and latency per accepted structured review. That denominator prevents a low-priced candidate with frequent rejection from looking better than it is.

How should small models, batch processing, prompt token counting, and JSON checks reduce LLM cost?

Start with task boundaries. “Review this change” is broad. “Summarize the changed behavior,” “classify the affected support area,” and “extract findings into a fixed JSON envelope” are narrower. A small model may clear those bounded steps while ambiguous security, concurrency, or data-loss findings move to a stronger review lane. This is routing, not a blanket replacement for GPT-4 or any other baseline.

Prompt token counting belongs before dispatch because it can enforce a context budget. Use the tokenizer that matches the candidate when one is available; otherwise treat the estimate as a routing hint, never an invoice. After completion, store the usage returned through the model adapter. Compare cost using the current input and output rates supplied by your own configuration, since a hard-coded article will age faster than a deployment file.

Batch processing is a queue policy. It helps only when a review can wait long enough to join a batch and the chosen execution service actually offers a favorable batch rate or throughput mode. An urgent patch should stay on the interactive lane. The catch is straightforward: batching can improve utilization while making tail latency worse, so a team with a tight response target may rationally keep interactive requests even when their per-request cost is higher.

Then validate the result in layers. A JSON parser answers whether the bytes are parseable. A schema check answers whether required fields and enums are present. Repository-aware checks answer whether a reported file exists and a line is within the changed range. None of those proves that the finding is insightful. A blinded human sample, scored with the same rubric across candidate lanes, supplies that missing signal.

Use a small decision table before tuning thresholds:

I'm not sure there is a universal “best” small model for this lane. There isn't enough information in a price table to settle it. Your mileage may vary with language mix, diff shape, schema complexity, and the cost of a missed finding; a replay set from the actual support repository resolves more uncertainty than a generic leaderboard.

Copy this TypeScript decision record before changing the model

The example below is deliberately vendor-neutral. The adapter returns usage and a structured candidate. Rates, latency limits, and quality floors are deployment inputs. The code does not pretend that one threshold works everywhere.

type Task = "summarize" | "classify" | "extract_json" | "review";
type Lane = "small-interactive" | "small-batch" | "strong-interactive";

type Finding = {
  severity: "low" | "medium" | "high";
  file: string;
  line: number;
  evidence: string;
};

type CandidateResult = {
  findings: Finding[];
  promptTokens: number;
  outputTokens: number;
  queueMs: number;
  generationMs: number;
};

type ReviewPolicy = {
  maxPromptTokens: number;
  maxTotalLatencyMs: number;
  changedLines: Map<string, Set<number>>;
};

type ReviewEvent = {
  reviewId: string;
  task: Task;
  lane: Lane;
  promptTokens: number;
  outputTokens: number;
  queueMs: number;
  generationMs: number;
  schemaValid: boolean;
  evidenceValid: boolean;
  accepted: boolean;
  escalationReason?: string;
};

function evidenceResolves(
  findings: Finding[],
  changedLines: Map<string, Set<number>>,
): boolean {
  return findings.every((finding) =>
    changedLines.get(finding.file)?.has(finding.line) === true &&
    finding.evidence.trim().length > 0
  );
}

function decide(
  reviewId: string,
  task: Task,
  lane: Lane,
  result: CandidateResult,
  policy: ReviewPolicy,
): ReviewEvent {
  const schemaValid = result.findings.every((finding) =>
    ["low", "medium", "high"].includes(finding.severity) &&
    finding.file.length > 0 &&
    Number.isInteger(finding.line)
  );
  const evidenceValid = evidenceResolves(result.findings, policy.changedLines);
  const totalLatencyMs = result.queueMs + result.generationMs;

  let escalationReason: string | undefined;
  if (result.promptTokens > policy.maxPromptTokens) {
    escalationReason = "prompt_budget";
  } else if (!schemaValid) {
    escalationReason = "schema_validation";
  } else if (!evidenceValid) {
    escalationReason = "evidence_validation";
  } else if (totalLatencyMs > policy.maxTotalLatencyMs) {
    escalationReason = "latency_budget";
  }

  return {
    reviewId,
    task,
    lane,
    promptTokens: result.promptTokens,
    outputTokens: result.outputTokens,
    queueMs: result.queueMs,
    generationMs: result.generationMs,
    schemaValid,
    evidenceValid,
    accepted: escalationReason === undefined,
    escalationReason,
  };
}

This record is intentionally boring. Good! It lets logs answer “why did this request escalate?” and lets metrics answer “which lane produces accepted reviews within the latency target?” without scraping prose. A counter can group accepted and escalated events by task and lane. Histograms can track prompt tokens, output tokens, queue time, and generation time. Alerts should follow user-visible failure budgets, such as a sustained fall in accepted-review rate or a rise in late reviews, rather than firing on every individual schema rejection.

Do not turn the event into a junk drawer. In particular, avoid a free-form error field as the primary dimension; bounded reason codes such as prompt_budget, schema_validation, evidence_validation, and latency_budget produce stable groups. Preserve the detailed diagnostic in a trace or restricted log when needed. This split keeps the high-cardinality evidence available without making every metric label expensive to aggregate.

The deployment loop is equally concrete. Consider an illustrative patch that changes ticket assignment and touches a routing rule, its test, and a response type. The bounded lane first summarizes the behavior, classifies the support area, and extracts findings. One finding points to the test's changed line and carries evidence; another points outside the diff. Deterministic validation accepts the first location and escalates the second, while the event record preserves both the token usage and time spent before escalation. A reviewer later scores a blinded sample that includes this review. Now the team can see whether the small lane reduced accepted-review cost or merely moved work into a second call. Repeat that flow over a fixed, versioned set of support-code changes. Compare accepted-review cost and latency, not raw call price. Release the route behind a percentage gate, watch the lane-level signals, and roll the route back when its quality floor or latency budget is crossed. No drama.

What about weak judges and slower batches?

The first objection is that model-based scoring can favor the same style of answer it generates. Correct. Keep deterministic checks for syntax and repository evidence, but don't promote a model lane solely because another model likes its prose. Human review is slower, so sample it: oversample escalations and high-risk changes, then retain a random slice of routine accepts to expose false confidence. The review rubric should score actionable evidence, severity calibration, duplicate findings, and missed material issues. Version that rubric alongside the replay set.

The second objection is that batching fights the latency goal. It can. Make the queue deadline explicit and route work that cannot wait to an interactive lane. Batch processing is not suitable when the support workflow needs an immediate review, arrival volume is too sparse to form useful groups, or the execution option has no relevant batch mode. Stick with interactive processing in those cases. Likewise, stick with a stronger model when the diff carries security-sensitive behavior, the structured finding needs substantial cross-file reasoning, or sampling shows the small lane below the quality floor.

Specialized ML systems offer a useful analogy without choosing a code-review vendor: reranking is documented as its own task, and speech recognition has a dedicated open-source model repository. The lesson is decomposition — a narrow interface can be evaluated on the outcome it owns. Those examples do not establish which model should review a support patch, and they do not replace repository-specific replay data.

One warning: don't optimize the visible token count while ignoring retries, rejected outputs, and reviewer labor. The routing decision is defensible only when the telemetry connects all of them to the final accepted review. Once that link exists, the team can lower LLM cost with small models where the evidence supports them, preserve a stronger lane where it doesn't, and treat batch processing as a latency-aware option rather than a slogan.

References

• https://docs.cohere.com/docs/rerank-overview
• https://github.com/openai/whisper