The arbitrage that wasn't

I build a scanner that looks for cross-venue arbitrage in prediction markets. The premise is simple enough to fit in one sentence: Polymarket, Kalshi, PredictIt and Limitless price the same real-world events independently, and when they disagree far enough you can buy YES on one venue and NO on the other for a combined cost under $1.00. Exactly one leg pays out $1.00 whichever way the world goes. It isn't a directional bet — you don't need to know who wins.

That's the theory. This post is about the part of the theory that turned out to be wrong, and what it cost to find out.

The number that shouldn't exist

A correctly matched hedge has a property you can check mechanically: exactly one leg pays. If you hold YES on one venue and NO on another for the same question, the two outcomes are complementary. One resolves true, one resolves false. Always.

So when I audited settled positions, this is what should have been impossible:

127 of 426 settled positions (29.8%) had both legs settle the same way.

Both won, or both lost. Nearly a third of them. That isn't a rounding error or a fee subtlety — it's a violation of the instrument's defining property. Something in my matching was pairing markets that were not actually complementary.

My first assumption was the obvious one: the matcher is sloppy, tighten the similarity threshold. That assumption was wrong, and it took a classification pass to see why.

Splitting the failures by cause

Instead of tuning a threshold, I split the 127 by a single question: do the two legs name the same subject?

The split was stark, and it separated two completely different problems that a similarity score had been quietly blending together. (23 named different subjects, 97 named the same subject, and 7 were too ambiguous to classify either way.)

Class A: different subjects — 23 positions

Genuine false matches. Two markets that read alike and are not the same event. The two that hurt most:

Cross-city weather. "Highest temperature in Atlanta 92-93" and "highest temperature in Miami 92-93" differ by exactly one token out of about twelve. IDF-weighted similarity scored them 0.65 — comfortably above threshold — and paired two different cities.

The insight is that a city is not one signal among many in a weather market. It is the market. So it can't be left to a bag-of-words score to weigh; it needs its own guard:

const locus = (() => {
  if (!/\b(temperature|rainfall|snowfall|weather)\b/.test(lower)) return null;
  const out = new Set();
  // "... in <City> on ..." / "... in <City> be ..." / "... in <City>?"
  const re = /\bin\s+([a-z][a-z .'-]{2,24}?)\s*(?:\bon\b|\bbe\b|\bbetween\b|,|\?|$)/g;
  let m;
  while ((m = re.exec(lower))) {
    m[1].split(/[^a-z]+/).filter((w) => w.length > 2).forEach((w) => out.add(w));
  }
  return out.size ? out : null;
})();

Then, in the scoring function, a disjoint pair of cities is killed rather than merely penalised:

if (a.locus && b.locus) {
  let shared = false;
  for (const w of a.locus) if (b.locus.has(w)) { shared = true; break; }
  if (!shared) score *= 0.12;
}

Note the comparison is overlap, not equality. Equality would reject "Portland" against "Portland Timbers" — a real pair. Overlap kills only genuinely disjoint names. That distinction shows up again and again in this codebase; matching on exact sets is almost always too brittle for titles written by different humans at different venues.

A cheap detail that cost real time: my first version of that regex terminated the lazy capture on a bare on, which quietly truncated "Boston" to "bost". The terminators have to be word-bounded — \bon\b|\bbe\b|\bbetween\b.

Opponent inversion. This one is my favourite, because both legs are correct in isolation and the pair is still nonsense. Venues name the same fixture two different ways:

• Polymarket: "Will Las Vegas win?"
• Kalshi: "Miami vs Las Vegas — Miami"

Buy YES on the first and NO on the second and you have… one directional bet placed twice. Both pay when Las Vegas wins. Both lose when Miami does. Every token matches. Similarity is high. It settles as a double win or a double loss depending purely on luck.

The fix parses which side each title actually backs — the tail after the dash in "A vs B — pick", otherwise the subject of "Will X win":

const vs = lower.match(/\bv(?:s\.?|ersus)?\b/);
const dash = lower.search(/[—–]|(?:\s-\s)/);
if (vs && dash > vs.index) {
  const w = words(lower.slice(dash).replace(/^[\s—–-]+/, ''));
  if (w.length) return new Set(w.slice(0, 2));
}

…and kills pairs backing disjoint sides.

Class B: same subject — 97 positions

Here's where it stopped being a matching problem.

These pairs were correctly matched. Same event, same wording, similarity around 0.80. The markets were genuinely identical. And both legs still settled the same way.

The reason: each venue resolves from its own source. Two venues can list "Will the high temperature in Denver exceed 90°F?" and read two different weather stations. One station reads 90.4°F, the other 89.8°F. Both legs settle YES, or both settle NO. The markets agree; the oracles disagree.

No amount of better matching fixes this. The similarity score is not wrong. The pair really is the same question. The hedge fails anyway, because the thing being hedged isn't a fact about the world — it's a fact about a measurement, and there are two measurements.

I've come to think of this as a distinct category: oracle risk. It is invisible to any text-similarity approach, because it lives entirely outside the text.

Since it can't be matched away, it gets priced as a risk instead:

const crossVenue = best.yesLeg.platform !== best.noLeg.platform;
const oracleRisk = crossVenue && /\b(temperature|rainfall|snowfall|weather)\b/.test(oracleText);
if (oracleRisk) riskFactors.push('weather market priced on two venues - each resolves from its own weather station, so both legs can settle the same way');
const matchRisk = (oracleRisk || sim < 0.6 || ...) ? 'high' : ...;

High match-risk positions never reach the recommended list. The pair still appears, correctly matched, with its warning attached — it just isn't presented as something to trade.

This also explained a distribution I'd been staring at without understanding: weather-heavy Markets failed at 24%, versus Sports at 9% and Entertainment at 6%. It wasn't that market questions were harder to match. It was that they were disproportionately resolved by instruments rather than by events.

The uncomfortable part

Here's what makes this genuinely awkward to write about honestly.

When both legs of a mispaired position settle YES, you get paid twice. My two largest single gains on record — +$1,105 and +$1,083 — came from exactly the mechanism that produced my largest losses. Same bug. Different coin flip.

So the temptation is obvious: publish the wins, call it arbitrage, quietly drop the rest. It would even be technically true that those trades made money.

It would also be cherry-picking the lucky half of a loss-making class. The correct headline is not the sum of the good outcomes; it's the scoped record after removing the class entirely. On the current live figures, that's 87 recommended positions settled, 74 of which paid out as designed, averaging +2.24% each, or +$1,657 per $1,000 committed — with the 13 that didn't behave as hedges named on the page rather than deleted from it.

A hedge that is correctly matched cannot lose. So a track record led by losses would misdescribe the instrument — but a track record with the failures deleted would misdescribe the engineering. Both numbers have to be visible. The full settled record is public, failures included.

Two testing lessons that nearly cost me

1. A passing test can prove nothing. My first tests for the opponent-inversion guard passed immediately. They were worthless. With the guard disabled, those cases still scored 0 — existing period and ordinal scope guards were already rejecting them for unrelated reasons, and the scorer returns 0 for "no match", so s < MIN_SIM passed trivially.

The rewritten tests use two titles with identical token sets — "Arsenal vs Chelsea — Arsenal" versus "— Chelsea" — so similarity, IDF and every other guard see identical input and the backed side is the only variable. Same side: 0.52. Opposite side: 0.00.

Always A/B a new guard by disabling it and confirming the case scores above threshold without it. Otherwise you're testing your other guards.

2. Verify a regex by calling it, never by reading it. I once wrote \b into a JS regex through a bash-heredoc → python pipeline. It emitted a literal backspace byte (0x08), not the two-character escape. The regex silently matched nothing. node --check passed. The file looked correct in every editor.

Detect with grep -c $'\x08' file. Better: never trust a regex you have only read.

A performance note, because ordering beat filtering

Unrelated to correctness but too good not to share. Matching took 61.5s per crawl, of which one venue pair — Polymarket × Kalshi, 10,528 × 41,733 markets — was 51.6s, or 84% of all matching CPU.

The inner loop looked like this:

let s = similarity(qtok, poolTok[i], idf) * dateFactor(...);

similarity() is the expensive token-set operation. dateFactor is 0 whenever two markets resolve more than 30 days apart. Since similarity() is bounded by 1 and every guard only scales it down, s <= dateFactor always holds — so computing the cheap factor first and bailing out is provably behaviour-preserving:

const df = dateFactorTs(qEnd, poolEnd[i]);
if (df === 0) continue;
const pm = poolMarkets[i];
let s = similarity(qtok, poolTok[i], idf) * df;

4.85× faster on that pair. Zero output difference — verified by A/B on a frozen snapshot, which mattered more than I expected: comparing across two live fetches proves nothing, because venue prices and dates drift between runs. My first comparison showed different hashes at identical pair counts and looked like a regression that wasn't there.

What didn't work, measured and rejected: pre-filtering the Kalshi pool to priced, liquid markets cut it from 41.7k to 11.6k rows (72% smaller) for only 13% speedup — and lost 486 real pairs. The cost was on the query side, not the indexed pool.

What I'd tell someone building this

1. When your data violates an invariant, don't tune a threshold — classify the violations. "Tighten the similarity score" would have suppressed some of Class A and none of Class B, while hiding the fact that two unrelated problems existed at all.
2. Some risks are not in the text. Two identical questions can resolve differently because they're measured differently. No NLP improvement reaches that. Price it, disclose it, or refuse the trade.
3. Match on overlap, not equality, whenever humans wrote the strings.
4. Your biggest wins deserve the same audit as your biggest losses. If you only investigate the losses, a broken mechanism that happens to be paying out will survive indefinitely.

The scanner is at provisum.ai if you want to see the output. The settled record — including the positions where the matching failed — is public.

Nothing here is financial advice. Past resolutions do not predict future ones.