My Paper Reader Answered Questions for Weeks. I Never Checked If It Was Right.

TL;DR

• I've been messing with a small project called Talkit, which reads research papers aloud and answers spoken questions about them. The answering half is retrieval-augmented generation, and this is what it took to find out whether it worked: grounding, generation, evals, tracing.
• There's no vector store. A paper fits in context, so the retrieval step is the whole paper plus the passage that was just read out.
• Evals run through DeepEval against ten questions about the Attention paper, scored four ways. Two of the questions have no answer in the paper.
• The app already emitted OpenTelemetry traces, so pointing them at Confident AI was four environment variables and one small code change. Making a span scoreable took a dozen attributes, and they stay off by default because they carry the question and the paper text.
• On the first real run, every answer scored 1.00 on faithfulness. The failures were elsewhere: a model that sometimes never answers, and one of my own goldens asking for more than its question did.
• The eval suite isn't in CI. It costs credits on every run, and the model under test sometimes returns no answer at all.

I've been chipping away at a small project called Talkit. It reads research papers aloud and lets you interrupt it. You say "wait, why do they scale that?", the narration stops, it answers in two or three spoken sentences, and then it picks up where it left off. That part works. I use it.

What I could not have told you, for weeks, was whether the answers were any good.

The evidence I had was two scripts. One uploads the Attention paper, asks a single question, and checks that an answer comes back. The other does the same in Hinglish. An answer that invented a BLEU score would have passed both. So would an answer that confidently explained the wrong paragraph. The endpoint returned 200 and the suite stayed green, and a green suite quietly becomes the thing you trust.

The other reason I put it off is that the RAG advice I kept reading didn't fit. Every guide opens with chunking and embeddings and retrieval metrics, and Talkit has no vector store at all: the paper fits in the model's context, so it sends the whole thing, which leaves half the standard eval playbook describing a component I don't have.

So I finally sat down and measured the thing I actually built. This post is what that took: pulling the answer path into one function, writing ten questions with known answers, four metrics that each catch a different failure, and span attributes that let a backend score answers as they happen. None of it is elaborate. It's the smallest setup that would tell me when an answer is wrong, and the first run turned up a failure mode that had been sitting there the whole time.

What the pipeline looks like

Here's the path a question takes:

1. The browser records the question and Sarvam's saaras:v3 transcribes it.
2. The server loads the paper text and the passage currently being read.
3. The model (sarvam-105b or Claude, whichever I've picked, on my own key) gets the whole paper, the current passage, recent turns, and the question.
4. The answer is stripped of anything a voice would read literally, and bulbul:v3 speaks it.

The question path across the top, the answer path and the trace below it.

Steps 2 and 3 are the RAG part, and the answer that comes out of them is what this post evaluates.

The retrieval step that isn't a vector search

Talkit has no chunking and no embeddings on the answer path, and that was a deliberate choice.

Uploads are capped at 60,000 characters. The Attention paper, after the reference list is cut, comes to 30,138. That fits in a model's context with plenty of room, so the model sees all of it.

The bigger reason is the kind of question I actually ask it. Halfway through a paper I'll say "why does that matter" or "what's this number". Those questions retrieve badly by similarity, because the words that matter ("that", "this") point at what was just read out, not at anything an embedding can match. So the current passage goes in as its own labelled block. That passage is the retrieval signal.

Every question sends the whole paper, so input tokens scale with paper length, not question complexity. The 60k cap also rules out books and long theses. If Talkit ever answers questions across a whole library, or takes documents past the cap, that's when I'd go add a vector store, and neither of those is true today.

Pulling the pipeline out of the route

The answer logic used to live inline in the /api/ask handler. An eval can't call a FastAPI route without a database, a session, and a stored key. And an eval that reimplements the prompt assembly measures a copy, which drifts the first time I edit the original.

So it moved into app/qa.py, and the route calls it:

async def answer_question(provider: TextProvider, lang: str, paper: str,
                          passage: str, question: str,
                          history: list[dict] | None = None,
                          user_id: str | None = None) -> str:
    with tracer.start_as_current_span("ask.answer") as span:
        messages = build_messages(paper, passage, history or [], question)
        raw = await provider.converse(answer_system(lang), messages,
                                      max_tokens=ANSWER_MAX_TOKENS)
        answer = speakable(raw, question)
        describe_llm_span(
            span, name="ask", model=provider.model, provider=provider.label,
            question=question.strip(), answer=answer,
            grounding=[passage, paper], user_id=user_id,
            metadata={"lang": lang, "history_turns": len(history or [])},
        )
        return answer

build_messages puts the paper and passage first, then up to six recent turns, then the question:

def build_messages(paper: str, passage: str, history: list[dict],
                   question: str) -> list[dict]:
    messages = [
        {"role": "user",
         "content": f"FULL PAPER:\n{paper}\n\nCURRENT PASSAGE:\n{passage}"},
        {"role": "assistant",
         "content": "Understood. I have the paper and I know where the listener is."},
    ]
    for turn in history[-HISTORY_TURNS:]:
        role = "user" if turn.get("role") == "user" else "assistant"
        messages.append({"role": role,
                         "content": str(turn.get("content", ""))[:HISTORY_TURN_CHARS]})
    messages.append({"role": "user", "content": question.strip()})
    return messages

And speakable removes markdown plus a restated question. The system prompt already forbids markdown and the model mostly complies, which isn't good enough for text that goes straight into a text-to-speech voice.

The refactor was pure motion: behavior didn't change, and the existing 85 tests still pass. I added six more for this module, which brings the offline suite to 91.

Pointing OpenTelemetry traces at Confident AI

Talkit already sends OpenTelemetry traces. Spans are created unconditionally, and with no exporter configured the global provider is a no-op. Picking a backend is environment variables, with no vendor SDK in the code.

I picked Confident AI because DeepEval, which I was already using for the evals, is theirs, so the eval runs and the traces land in the same project. Langfuse, Phoenix and others accept OpenTelemetry too, and everything up to this point would work with any of them. Pointing it at Confident AI:

OTEL_EXPORTER_OTLP_ENDPOINT=https://otel.confident-ai.com
OTEL_EXPORTER_OTLP_HEADERS=x-confident-api-key=<key>
OTEL_LOGS_EXPORTER=none
OTEL_RESOURCE_ATTRIBUTES=confident.trace.environment=production

The HTTP exporter appends /v1/traces to the endpoint, so the base URL is enough. The environment label rides on the standard resource attribute variable, which the SDK already reads.

The third line needed a code change. Talkit ships logs through the same exporter so each log record carries its trace and span id. Confident AI's OpenTelemetry docs say the endpoint takes traces and not logs, so the logs have to be turned off separately. OTEL_LOGS_EXPORTER is the standard variable for this, so the setup now respects it:

if ENDPOINT and OTEL_LOGS_EXPORTER != "none":
    logs = LoggerProvider(resource=resource)
    logs.add_log_record_processor(BatchLogRecordProcessor(OTLPLogExporter()))
    set_logger_provider(logs)
    logging.getLogger().addHandler(LoggingHandler(logger_provider=logs))

At this point traces arrive, but they're only timings. A trace backend can show that ask.answer took four seconds. It can't tell whether the answer was grounded in the paper.

Making a span scoreable

To score an answer, Confident AI needs to know which span is the model call and what went in and came out. It reads that from span attributes in its own namespace. Talkit writes them in one function in app/telemetry.py:

def describe_llm_span(span, *, name: str, model: str | None, provider: str,
                      question: str, answer: str, grounding: list[str],
                      user_id: str | None = None, metadata: dict | None = None,
                      content: bool | None = None,
                      metric_collection: str | None = None) -> None:
    content = TRACE_CONTENT if content is None else content
    collection = TRACE_METRIC_COLLECTION if metric_collection is None else metric_collection

    span.set_attributes({
        "confident.span.type": "llm",
        "confident.trace.name": name,
        "gen_ai.request.model": model or "unknown",
        "gen_ai.provider.name": provider.lower(),
    })
    if user_id:
        span.set_attribute("confident.trace.user_id", user_id)
    if metadata:
        span.set_attribute("confident.span.metadata", json.dumps(metadata))

    if not content:
        return
    span.set_attributes({
        "confident.span.input": json.dumps(question),
        "confident.span.output": json.dumps(answer),
        "confident.span.retrieval_context": json.dumps(grounding),
        "confident.trace.input": json.dumps(question),
        "confident.trace.output": json.dumps(answer),
    })
    if collection:
        span.set_attribute("confident.span.metric_collection", collection)

A few things here cost me more time than the code suggests.

Text values are JSON-encoded. The convention expects "\"Why scale it?\"", not "Why scale it?", and a list of passages becomes a JSON array string. Before writing this I ran Confident AI's own confident-trace package against an in-memory exporter and read what it set, so the shapes match what their SDK emits. The collection name is the exception: it's a plain string. A backend accepts a wrongly encoded attribute without complaint and just shows an empty panel, so the tests assert the encoding directly.

The grounding is the whole paper. retrieval_context is [passage, paper], because that's what the model saw. If I sent only the passage, the faithfulness metric would flag every true claim drawn from elsewhere in the paper as unsupported.

Content is off unless you turn it on. Scoring needs the question, the answer, and the paper attached to the span and exported. That's my reading and my questions leaving for a third party. So TRACE_CONTENT=true is an explicit switch. Without it, the span carries the span type, model name, and metadata, and nothing a person wrote.

Why attributes and not the SDK

There is an SDK for this, confident-trace, built on OpenTelemetry. It auto-instruments frameworks like LangChain, LlamaIndex and Pydantic AI, which would be the argument for using it.

Talkit calls Sarvam over httpx and Claude through the Anthropic SDK directly, so there is nothing for it to auto-instrument. It would add a dependency and I would still set the input, output and grounding by hand. Writing the attributes myself keeps "no vendor SDK" true, and the ten confident.* names above are the entire vendor-specific surface: swap backends and the spans, timings and gen_ai.* attributes go with you. The cost is that a vendor's attribute names now sit in the code, and if they change, that function changes with them.

The golden dataset: ten questions about one paper

The live suites in Talkit already use arXiv 1706.03762v7, Attention Is All You Need, so the evals do too. The goldens live in evals/goldens.json. Each one has a question, a phrase that locates the current passage, a language, and an expected answer:

{
  "question": "Why do they divide by the square root of d k here?",
  "passage_contains": "particular attention",
  "lang": "en",
  "expected_output": "For large values of d k the dot products grow large in magnitude, which pushes the softmax into regions with extremely small gradients. Scaling by one over the square root of d k counteracts that."
}

That one does double duty. It tests the decision to send the whole paper: the current passage is the one that introduces scaled dot-product attention, while the reason for the scaling comes later on, so an answer that only used that passage would miss it.

The passage is located by searching the chunks for that phrase:

def locate(chunks: list[str], needle: str) -> str:
    flat = lambda s: re.sub(r"\s+", " ", s).lower()
    for chunk in chunks:
        if needle.lower() in flat(chunk):
            return chunk
    raise SystemExit(f"no passage contains {needle!r}; did chunking change?")

An index would silently point at a different passage the first time chunking changes. A missing phrase fails loudly. I checked every phrase against the extracted text before writing the expected answers, and each one matches a passage among the paper's 46.

What the ten cover:

• Six factual questions with answers stated in the paper: layers, heads, the positional encoding choice, training time, label smoothing, BLEU.
• Two refusals. "How much GPU memory did the big model need?" isn't in the paper, and neither is the weather where the authors work. The system prompt says to say so in one sentence rather than fill the gap from general knowledge. These are the only way to check that it does.
• One Hinglish question, "Encoder mein kitne layers hain?", because Hinglish is a real language option in Talkit with its own prompt, and it can break independently of English.
• The scaling question above.

Ten questions doesn't cover much of anything. I'll add a golden every time a real answer comes back wrong.

Four metrics, each for a different failure

def metrics_for(model) -> list:
    return [
        FaithfulnessMetric(model=model, threshold=0.8),
        AnswerRelevancyMetric(model=model, threshold=0.7),
        GEval(
            name="Correct",
            model=model,
            threshold=0.7,
            evaluation_params=[SingleTurnParams.INPUT,
                               SingleTurnParams.ACTUAL_OUTPUT,
                               SingleTurnParams.EXPECTED_OUTPUT],
            evaluation_steps=[
                "Check that every fact in the expected output appears in the actual output. Wording may differ.",
                "Penalise any number, name or claim in the actual output that contradicts the expected output.",
                "If the expected output says the paper does not address the question, the actual output must say so and must not supply an answer from general knowledge.",
                "The actual output must be in the same language and script as the expected output.",
            ],
        ),
        GEval(
            name="Speakable",
            model=model,
            threshold=0.7,
            evaluation_params=[SingleTurnParams.ACTUAL_OUTPUT],
            evaluation_steps=[
                "The output will be read aloud by a text-to-speech voice to someone with no screen.",
                "It should be two to four sentences of plain prose.",
                "Penalise lists, headings, markdown, LaTeX, and symbols a voice would read literally, such as square root signs, subscripts or underscores.",
                "Numbers and variable names should be written the way a person would say them.",
            ],
        ),
    ]

Faithfulness extracts the claims in the answer and checks each against the grounding, which is how it catches invented numbers.

Answer relevancy catches an accurate answer to a different question, a real risk when the question is "what does this mean" and "this" could be several things.

Correct compares against the expected answer, and it's the metric that scores the refusals. Faithfulness can't: "the paper doesn't say" makes no claims to check.

Speakable is the one no generic metric covers. A Talkit answer is heard, not read. A correct answer containing √dk hands the text-to-speech voice a symbol where it needs words, which is why the prompt asks for spoken forms and this metric checks that it got them.

Anything code can check, code checks first, and prints before any judge weighs in:

def mechanical_failures(answer: str, lang: str) -> list[str]:
    failures = []
    sentences = [s for s in re.split(r"(?<=[.!?])\s+", answer) if s.strip()]
    if not 1 <= len(sentences) <= 5:
        failures.append(f"{len(sentences)} sentences")
    if lang == "hinglish" and DEVANAGARI.search(answer):
        failures.append("Devanagari in a Hinglish answer")
    if re.search(r"[√∈_\\$]", answer):
        failures.append("unspeakable symbol")
    return failures

Hinglish is romanised Hindi, and both Hinglish prompts forbid Devanagari outright because a model asked for it can drift into Hindi script, which a regex catches without any judge weighing in.

A judge from a different family

Every metric above is a model grading a model. Zheng et al. (2023) documented that LLM judges show self-enhancement bias: they favor answers written by the same model. So the judge defaults to a different family from the one answering. Talkit's default text model is sarvam-105b, and the default judge is claude-sonnet-5. Evaluating the Claude path instead trips a warning that suggests an OpenAI judge:

if args.provider == "claude" and args.judge.startswith("claude"):
    print("warning: Claude is judging Claude. Pass --judge gpt-5.4 for a "
          "judge from a different family.")

The Claude judge broke on its first run. DeepEval's Anthropic wrapper defaults to 1,024 output tokens. Faithfulness starts by listing every factual statement in the grounding, and the grounding here is a whole paper. That list ran past 1,024 tokens, came back as truncated JSON, and DeepEval reported "Evaluation LLM outputted an invalid JSON. Please use a better evaluation model." The fix was the budget:

def judge(name: str):
    if name.startswith("claude"):
        return AnthropicModel(model=name, generation_kwargs={"max_tokens": 16000})
    return OpenAIModel(model=name)

Next time a judge hands back broken JSON over long context I'll check the token limit before I blame the model.

Running it

Four lines, run by hand when I've changed something worth checking:

pip install -r evals/requirements.txt
curl -L -o attention.pdf https://arxiv.org/pdf/1706.03762v7
export SARVAM_API_KEY=... OPENAI_API_KEY=...
export CONFIDENT_API_KEY=...
python evals/run.py --pdf attention.pdf --judge gpt-5.4

The script loads the paper exactly as an upload does (extract, cut the back matter, cap at 60k), answers all ten goldens through answer_question, runs the mechanical checks, then hands the test cases to DeepEval. With CONFIDENT_API_KEY set, the run uploads to Confident AI tagged with the provider, the model, the judge, and a short hash of the answer prompt. That hash is what makes prompt changes comparable, since two runs with different hashes can be put side by side against the same goldens.

Evals have their own requirements file, which includes the app's requirements plus deepeval==4.2.3. The Docker image never installs DeepEval. One side effect: DeepEval needs python-dotenv 1.1.1 or later, so the app's pin moved up from 1.0.1.

Passages in the eval come from the extracted text. The app narrates a cleaned read-aloud script instead, and cleaning a paper takes minutes of paid API calls, which isn't what this suite measures. Wording differs slightly from what I actually hear, and I've accepted that.

What the first run found

I ran it with sarvam-105b answering and gpt-5.4 judging:

Nine of nine, because one question never got an answer.

Sometimes there's no answer at all. sarvam-105b is a reasoning model, and on one question it spent the entire 3,000-token budget reasoning and never wrote a word. The provider raises ChatBudgetExceeded with finish_reason=length. I ran the suite four times. Three runs had exactly one unanswered question (the BLEU question twice, the attention-heads question once) and one run answered all ten. The failure is intermittent and doesn't track any particular question. The same thing happened through the app itself, where "How many layers are in the encoder?" came back as a 502. Before evals, the only live test asked one question, so whether it passed depended on the run. The first version of the eval script crashed on it and threw away nine paid answers, so it now records an unanswered question as a failed case with the reason attached.

Faithfulness didn't discriminate. Every answer scored 1.00. The answers stayed inside the paper, including the refusals. Asked how much GPU memory the big model needed, it said the paper doesn't say, and added that the model has 213 million parameters, which is in Table 3. That's a good result, but it means faithfulness alone would have told me nothing, and the metric that actually moved was Correct.

The one Correct failure was my golden's fault. The Hinglish question asks "Encoder mein kitne layers hain?", how many layers are in the encoder. The answer said six. My expected output also listed the two sub-layers, which the question never asked for, and the judge docked it to 0.65 for leaving them out. The judge's reason spelled it out. The golden needs fixing, and the prompt can stay as it is.

Speakable let one thing through that I'd call a miss: the answer to the scaling question writes "dk" for d k, which a voice reads as a two-letter word. The judge gave it 0.83, comfortably above threshold, and the mechanical check doesn't look for it either. Later answers to the same question through the app did it again. That's the next check to add in code, since a regex settles it better than a judge.

One eval run: the nine questions that got answered, four metrics each, and the hyperparameters that produced them.

The judge's written reason names the detail my expected answer asked for and the question never did.

Why it isn't in CI

Talkit's offline checks are pyflakes, 91 tests, and a Node script that checks the voice command matcher against the shipped HTML, and none of them cost anything.

This suite costs credits on every run: ten answers, then four metrics per answer, and each metric makes several judge calls. And as those runs showed, the model under test sometimes doesn't answer at all. A hard gate would turn that into a flaky build I'd just rerun until it went green.

So it runs by hand, like the two live suites that exercise Sarvam's speech APIs. The script exits non-zero if any case fails a metric, a mechanical check, or gets no answer, so it could run in a scheduled job later. For now it's a tool I run before changing a prompt or a model.

Scoring answers as they happen, with a metric collection

Offline evals only know the questions I thought of. Actually using the thing turns up the ones I didn't.

Confident AI can score incoming traces with a metric collection: a named set of metrics defined in the project. Talkit reads the name from the environment:

TRACE_CONTENT=true
TRACE_METRIC_COLLECTION=talkit-answers

With both set, every ask.answer span carries the question, the answer, the grounding, and the collection name, and Confident AI runs the collection's metrics against it. Faithfulness and answer relevancy both work here. Correctness doesn't, because a question asked in the moment has no expected answer written for it. Correctness still comes from goldens. When a live answer scores badly, it becomes a golden.

One trap when creating the collection: Confident AI offers multi-turn metrics like Turn Faithfulness and Turn Relevancy alongside the single-turn ones. A collection of turn metrics has nothing to score on these spans. Talkit's answer span is a single turn, one question and one answer with its grounding, so the collection needs the single-turn Faithfulness and Answer Relevancy.

The other thing that cost me an evening: online evaluation needs a judge model of your own. Confident AI includes 100 traces of online evaluation, and after that the trace still arrives, still shows the collection name on the span, and simply is not scored until you configure an evaluation model in the project. The span looked correctly wired for a while before I noticed the banner saying so.

With a judge configured, the first scored answer came back at 1.00 on both faithfulness and answer relevancy, with a written reason on each, against the same passage and paper the model had been given. The scores in the collection default to a threshold of 0.5, while the offline suite uses 0.8 for faithfulness and 0.7 for relevancy, so the same answer can pass in one place and fail in the other. I haven't reconciled them yet.

One answer from the running app, scored on arrival against the passage and paper the model was given.

Expected Output is empty, which is the whole reason correctness can't be scored here. Note the 0.5 thresholds against the suite's 0.8 and 0.7.

When one does score badly, the span has what I need to start: the model and provider, the language, how many history turns were in play, and the exact grounding. The usual first question is whether the problem is in the current passage or the answer. Unlike a vector-search RAG system, there's no "wrong chunks retrieved" branch to rule out, since the model had the whole paper.

What else turned up

Running the whole thing locally, upload to answer, surfaced two problems outside the answer path.

Cleaning falls back more than I knew. Uploading the Attention paper through Sarvam took 837 seconds, and 14 of its 28 cleaning batches fell back to raw text. The log shows the same cause 52 times: a batch overran the token budget and was halved, until it hit the split limit and kept the raw text. That's the same reasoning-budget problem as the missing answers, on the cleaning side. A fallback is recorded and never surfaced, so I just hear citation markers read out with no explanation. It was already on Talkit's list of known gaps. Now it has a number.

The title is wrong. Talkit's title extractor reads the Attention paper's title as "Provided proper attribution is provided, Google hereby grants permission to", which is the permission banner at the top of the arXiv version.

Neither is fixed. I only ran into them because I was reading logs and traces for a different reason.