One AI Answer, Eight Brands: Designing a Benchmark Without Multiplying the Evidence

I recently ran a small China AI benchmark for eight luxury-jewelry brands.

The most interesting result was not a platform ranking.

It was a disagreement between two kinds of visibility.

Piaget appeared in all four answers about brands with verifiable official China channels. It appeared in none of the four answers recommending brands for wedding jewelry.

That is a tiny sample, so it is not evidence that Piaget has a stable visibility problem.

It is evidence that I needed more than one metric.

It also forced me to separate two units that are often quietly conflated in AI visibility datasets:

1. the answer collected from the model;
2. the brand-level judgment derived from that answer.

That distinction changed the collector, the denominator and what I was willing to publish.

The tempting but wrong collection design

The cohort contained eight brands:

• Cartier
• Tiffany & Co.
• Bvlgari
• Van Cleef & Arpels
• Chaumet
• Boucheron
• Piaget
• De Beers Jewellers

I had three neutral buyer questions, two API surfaces and two replicates.

A naive brand-expanded collector could have produced:

8 brands
× 3 questions
× 2 surfaces
× 2 replicates
= 96 provider calls

That would look like a larger dataset.

It would also change the experiment.

If I asked the same neutral question eight times—once for every target brand—I would collect eight different generated answers and then compare brands across different evidence.

Randomness, provider conditions and answer variation would be mixed with brand treatment.

The larger row count would not mean I had more independent evidence about the same answer. It would mean I had asked for more answers.

So I used an answer-once design.

3 questions
× 2 surfaces
× 2 replicates
= 12 valid raw answers

Each raw answer was then evaluated against the eight brands:

12 answers
× 8 brands
= 96 answer-brand cells

The dataset still contains 96 brand-level observations, but it does not pretend that 96 model responses were collected.

Collection unit and analysis unit

The collection unit was:

platform_surface_id
× question_id
× replicate_id

The derived analysis unit was:

answer_id
× target_brand_id

A simplified record relationship looks like this:

Answer
├── answer_id
├── platform_surface_id
├── question_id
├── replicate_id
├── raw_answer
├── validity
└── AnswerBrandCell[8]
    ├── target_brand_id
    ├── mentioned
    ├── recommended
    ├── official_channel_asserted
    └── human_review_state

This matters because Mention, Recommendation and channel accuracy are not properties of the API request alone.

They are judgments about a particular brand inside a particular answer.

The raw answer should remain immutable. Brand-level labels can be reviewed, corrected and replayed without recollecting the model output.

Invalid responses are attempts, not negative answers

A third API surface was planned for the pre-wave.

All six requests returned an account-state 403.

Those rows belong in the collection record because the attempts happened under a declared instrument. They do not belong in any answer-content denominator.

The correct accounting was:

planned answer cells = 18
valid answers        = 12
invalid by reason    = 6

It was not:

valid answers        = 12
brand absent         = 6

An error row proves that a collection attempt occurred. It contains no evidence about whether Cartier, Piaget or any other brand was mentioned or recommended.

I also retained two earlier Doubao timeouts from an instrument-configuration run. They remain in the audit trail but sit outside the final planned set because the collector had not carried forward the reasoning-mode configuration used during access testing.

Keeping failures does not require pretending every failure belongs to the published denominator.

One pooled visibility score would have hidden the result

The three questions represented different buyer decisions:

1. Which high-end brands are worth considering for wedding jewelry?
2. How should a buyer compare a brand boutique with a daigou purchase?
3. Which international jewelry brands have verifiable official China channels?

The aggregate results were:

Those denominators are not interchangeable.

The recommendation denominator contains the eight brand cells from the four wedding answers.

The risk-question denominator contains the eight brand cells from a different four answers.

The route-match denominator contains only exact domain assertions that were eligible for truth review.

Combining them into one “AI visibility score” would make the output simpler and the diagnosis worse.

The Piaget row demonstrates why:

The brand was recognized when the task was channel verification. It was absent when the task was open-category recommendation.

A domain or entity project might improve verification while doing nothing to answer why the brand did not enter the shortlist.

The inverse problem also appeared:

A brand can enter consideration while its verification route is reproduced less consistently.

These are different business problems, with different owners and different retest questions.

Store membership, not only the percentage

A count can be reproducible while the population is wrong.

Two filters can both return 32 rows and still return different sets of 32 rows.

For every published metric, I therefore stored the exact denominator and numerator memberships.

A simplified manifest looks like this:

{
  "metric_id": "wedding_recommendation",
  "unit": "answer_brand_cell",
  "eligible_predicate": {
    "question_id": "JW-CORE-01",
    "answer_validity": "VALID"
  },
  "denominator": {
    "member_ids": [
      "answer-001::cartier",
      "answer-001::tiffany"
    ],
    "sha256": "..."
  },
  "numerator": {
    "predicate": "human_recommendation_label == POSITIVE_SHORTLIST",
    "member_ids": [
      "answer-001::cartier"
    ],
    "sha256": "..."
  },
  "excluded_by_reason": {
    "invalid_answer": [],
    "not_eligible_question": []
  }
}

The real manifest contains the full member sets rather than the abbreviated example above.

A separate verifier rebuilds those sets from the stored row states and checks invariants such as:

assert(isSubset(numeratorIds, denominatorIds));
assert(plannedCount === validCount + invalidByReasonCount);
assert(intersection(errorRowIds, denominatorIds).size === 0);
assert(hash(rebuiltDenominatorIds) === storedDenominatorHash);
assert(hash(rebuiltNumeratorIds) === storedNumeratorHash);

The important guarantee is not merely:

21 / 32 can be recalculated

It is:

the same 21 cells and the same 32 cells can be reconstructed

That catches two implementations producing the same percentage from different observations.

Recommendation required human context

A string match was sufficient for a first-pass entity mention.

It was not sufficient for Recommendation.

A brand name can appear because the answer:

• recommends it;
• rejects it;
• compares it;
• repeats the prompt;
• cites a warning;
• says it is unsuitable;
• lists it as an example without putting it on the shortlist.

Every counted recommendation was therefore reviewed in the full answer and assigned to the target brand cell.

The rule was deliberately narrower than positive sentiment:

Count the brand only when the answer explicitly recommends it
or places it on a positive shortlist for the buyer decision.

Comparison, exclusion and incidental mention did not qualify.

This is slower than a regex. For a 96-cell exploratory study, it was still practical and materially safer.

Retrieval off means Citation is not observable

The two valid surfaces were retrieval-off APIs.

The answers can be studied as dated model outputs. They cannot establish which sources influenced the result.

Even if an answer prints a domain, that does not prove that the model retrieved or used the page during the request.

So the study reports:

Supported target Citation Rate: NOT_OBSERVABLE

It does not convert missing citation evidence into zero.

That distinction is important for longitudinal work. If a later wave uses a consumer search surface with observable retrieval and attributable citations, it should become a separate series rather than silently extending the retrieval-off API series.

The technical design changed the business question

The implementation lesson is straightforward:

• collect each neutral answer once;
• preserve the raw response;
• derive brand-level cells separately;
• retain invalid attempts by reason;
• freeze eligible populations by metric;
• store exact numerator and denominator memberships;
• make unsupported measurements NOT_OBSERVABLE, not zero.

The more interesting consequence is that a visibility audit should not end with one score.

For a luxury brand, I now want at least these separate layers:

Entity recognition
→ Open-category consideration
→ Positive recommendation
→ Official-channel accuracy
→ Supported source, when observable

A brand can fail at any transition.

Knowing which transition failed is more useful than knowing that an overall score moved from 42 to 47.

What the next wave needs

This pre-wave contains only four answer opportunities per brand for each task. It is enough to find a measurement problem, not enough to establish a stable brand position.

A formal next wave would need:

• fixed consumer and API surface series;
• a larger but stable buyer-question panel;
• unchanged brand and alias registries;
• a dated channel and policy truth table;
• repeated comparable cells;
• the same metric membership contract;
• separate reporting by buyer decision;
• no platform winner unless each platform cell is complete and comparable.

I would deepen the buyer journey before adding dozens of brands.

For jewelry, that means questions around wedding purchases, anniversary gifts, diamond selection, high-jewelry commissions, boutique-versus-daigou risk and mainland after-sales confidence.

The goal is not to manufacture a larger dataset.

It is to find out where a brand moves from being known, to being considered, to being recommended, to being correctly verified.

In this small pre-wave, Piaget made the final distinction visible: the models could place the brand on the official China channel map, but they did not place it on the wedding shortlist.

That is not a conclusion about the brand.

It is a much better next question than “What is its AI visibility score?”