Short answer: for edtech moderation reports, start with an embeddings classifier for routine labels, send uncertain cases to a zero-shot LLM, and reserve reranking for reports whose top candidate labels are too close; record every stage against the tenant that caused the work. This is usually a better alternative to fine-tuning than forcing one model to handle every report, because it keeps the implementation understandable while making per-tenant cost visible.
The important constraint isn't raw model quality in isolation. A school with repetitive attendance-related reports and a tutoring marketplace with long free-text safety complaints can consume very different amounts of inference. A single blended monthly total hides that difference and makes a cheap system surprisingly hard to operate.
Keep the bill explainable.
What should replace fine-tuning for cheap, simple support ticket tagging?
An embeddings classifier, a zero-shot LLM, and a reranker solve different parts of the tagging problem. Treating them as interchangeable produces misleading comparisons. For a moderation queue, the classifier answers, "Which known label is this report most similar to?" The zero-shot model interprets the report against written policy labels without task-specific training. The reranker takes a small candidate set and puts the most relevant label first. The practical default is a cascade: embed the report, compare it with approved label examples, and accept the result only when the score and margin pass thresholds chosen on a held-out evaluation set. If the classifier is unsure, ask the zero-shot model to select from the same versioned taxonomy. If two or three policy labels remain close, rerank those candidates rather than the whole catalog. Human review remains the final destination for low-confidence or high-impact reports. This is not a universal winner. An embeddings-first design is a poor fit when labels depend on subtle multi-step policy reasoning rather than semantic similarity. A zero-shot-first path may be appropriate when the taxonomy changes every day and traffic is low enough that repeated generation work is acceptable. A reranker is unnecessary when there are only a few sharply distinct labels. And if labeled volume becomes large, stable, and representative across tenants, fine-tuning may deserve another evaluation. There is another edge: tenant isolation. Never build one tenant's examples into another tenant's candidate pool unless the data policy explicitly allows it. Moderation text can contain personal data, prompt injection attempts, and quoted abusive content. OWASP's guidance for LLM applications is a useful reminder that model input is untrusted input — classification does not make it safe. Keep authorization, retention, output validation, and human escalation outside the model call.
Tenant boundaries matter.
Cost visibility starts before model selection
Per-tenant cost cannot be reconstructed reliably from a provider invoice after the fact. The application needs its own usage ledger at the boundary where work is scheduled. Each classification attempt should carry tenant_id, report_id, taxonomy version, stage, model or index version, input units, output units, latency, decision, and whether a human review followed. Store money as a decimal value or integer minor units, never binary floating point.
The ledger should distinguish attempted work from accepted decisions. Retries still cost something. Cache hits may avoid a model call but still use storage and lookup capacity. A report that passes through embedding, zero-shot classification, and reranking must create three stage records linked by one trace identifier; otherwise the expensive tenants look identical to the easy ones.
Here's a compact Python sketch. The thresholds are examples, not claimed universal values, and the rates arrive through deployment configuration rather than being hard-coded as vendor prices.
from dataclasses import dataclass
from decimal import Decimal
from typing import Protocol
@dataclass(frozen=True)
class Decision:
label: str
confidence: float
stage: str
class UsageLedger(Protocol):
def record(
self, *, tenant_id: str, report_id: str, stage: str,
input_units: int, output_units: int, cost: Decimal
) -> None: ...
def classify_report(
*, tenant_id: str, report_id: str, text: str, taxonomy: list[str],
services, ledger: UsageLedger, rates: dict[str, Decimal]
) -> Decision:
candidates, embed_units = services.embedding_candidates(text, taxonomy)
ledger.record(
tenant_id=tenant_id, report_id=report_id, stage="embedding",
input_units=embed_units, output_units=0,
cost=Decimal(embed_units) * rates["embedding_unit"],
)
top, runner_up = candidates[0], candidates[1]
if top.score >= 0.82 and top.score - runner_up.score >= 0.10:
return Decision(top.label, top.score, "embedding")
label, confidence, input_units, output_units = services.zero_shot(
text, taxonomy
)
ledger.record(
tenant_id=tenant_id, report_id=report_id, stage="zero_shot",
input_units=input_units, output_units=output_units,
cost=(Decimal(input_units) * rates["llm_input_unit"]
+ Decimal(output_units) * rates["llm_output_unit"]),
)
if confidence >= 0.75:
return Decision(label, confidence, "zero_shot")
label, score, rerank_units = services.rerank(text, candidates[:3])
ledger.record(
tenant_id=tenant_id, report_id=report_id, stage="rerank",
input_units=rerank_units, output_units=0,
cost=Decimal(rerank_units) * rates["rerank_unit"],
)
return Decision(label, score, "rerank")
Don't copy those thresholds into production unchanged. Calibrate them per taxonomy version, then decide whether tenants share thresholds based on measured error distributions. A global threshold is operationally simple, but it can make one tenant subsidize another tenant's ambiguity. Your mileage may vary, especially when one customer writes terse reports and another pastes full conversation transcripts.
Compare the approaches by failure mode, not demos
A demo dataset often makes all three approaches look competent. The differences appear in the tails: newly introduced policy labels, nearly synonymous categories, multilingual text, empty reports, copied email headers, adversarial instructions, and very long context. This is familiar territory to anyone who has designed OTP or notification flows. The happy path is short; delivery gaps, rate limits, and abuse controls determine the architecture.
| Approach | Best fit | Main operational cost | Failure to watch | Prefer another path when |
|---|---|---|---|---|
| Embeddings classifier | Stable labels with representative examples | Embedding writes, lookups, and index maintenance | Similar labels collapse into close neighbors | Decisions require policy reasoning not present in examples |
| Zero-shot LLM | New or frequently edited label descriptions | Input and output units on each uncached call | Output drifts outside the allowed schema | Volume is repetitive and a validated similarity rule handles it |
| Reranker | A small, plausible candidate set needs ordering | Scoring each report-candidate pair | Bad retrieval excludes the correct label before ranking | The label set is already tiny and distinct |
| Fine-tuned classifier | Stable task with enough governed labeled data | Dataset curation, training, evaluation, and rollout | Taxonomy changes invalidate learned behavior | Labels or policy definitions change faster than retraining |
Evaluate at the tenant level, not only across the pooled corpus. Track macro F1 or per-label precision and recall where they match the moderation risk, but also inspect abstention rate, escalation rate, cost per accepted tag, and cost per reviewed report for each tenant. A low average can conceal a tenant whose reports always traverse the full cascade.
I'm not sure a single accuracy aggregate can settle the decision for a moderation team. The missing evidence is the cost of each error class: misrouting spam is not equivalent to under-prioritizing a credible safety report. Define that loss with policy owners before tuning a threshold. For sensitive labels, choose abstention. Fast and wrong is still wrong.
For an embeddings store, PostgreSQL with the pgvector extension is one standards-friendly implementation option: it keeps vectors alongside relational tenant and taxonomy metadata and supports exact and approximate nearest-neighbor search. It isn't automatically the right choice. Stick with a dedicated index when scale, filtering behavior, or operational ownership makes that boundary clearer, and keep the classifier interface independent so storage selection does not leak into policy code.
Roll out without losing the audit trail
Begin in shadow mode: generate tags, costs, and reasons without changing the human queue. Compare decisions against reviewer outcomes, split metrics by tenant and label, and test malformed, multilingual, duplicated, and prompt-like reports. A result that cannot be tied to a taxonomy version should fail closed into review.
Then enable automatic tagging for a narrow set of low-impact labels. Raise coverage only after the per-tenant ledger reconciles with provider usage and the abstention path behaves as designed. Version example sets and label descriptions together, because changing either one changes the classifier even when application code stays still.
Watch for HTTP 429 responses at external model boundaries and apply bounded retries with jitter, but charge each attempt to the initiating tenant. Never retry an ambiguous classification merely because the answer was inconvenient. Rate limiting is transport behavior; uncertainty is a decision outcome.
The migration stays reversible when every stage implements the same typed contract and emits the same audit fields. You can replace the embedding store, zero-shot model, or reranker without rewriting tenant accounting or moderation policy. That separation — policy, inference, and billing evidence — matters more than squeezing the entire system into one clever model call.
