Connecting an LLM to Your Data Is the 21% Solution.

I hear the same question in almost every early conversation: why do I need a platform when I can point Claude at Postgres and start asking questions?

It is a fair question. The connectors are real. Claude wires into Snowflake and Postgres, ChatGPT reaches warehouses over MCP bridges, Gemini sits inside BigQuery. The demo is genuinely impressive.

We also now have public numbers on what happens after the demo.

The numbers

Anthropic measured it on their own analytics workload: "Without skills, Claude's ability to answer analytics questions accurately didn't exceed 21% on our evals." Same model you would connect to your database. Their data, their questions, 21%.

The benchmarks agree. Spider 2.0 is built from real enterprise warehouse workflows: thousand-column schemas, hundred-line queries, multiple SQL dialects. GPT-4o scores 10.1% on it. o1-preview reaches 17.1%. The same models clear 85% on the older academic benchmark, which is why the demo feels like magic and the fourth week does not.

BIRD explains the collapse. Every BIRD question ships with a hand-written hint explaining what the fields mean. Remove the hints and GPT-4 falls from 54.89% to 34.88%. Your users do not write hints. They ask "how many active customers do we have" and assume the model knows what active means. It does not. It picks one of your four dozen candidate columns and sums with confidence.

That is the dangerous part. A wrong number does not look wrong. Anthropic names it the silent failure and admits their hardened stack reduces it rather than eliminating it.

What the 74 points cost

Anthropic closed the gap from 21% to above 95%. Not with a bigger model, and not with more data access: they gave the agent grep access to thousands of their own correct past queries and accuracy moved less than one point.

What worked was structure. Governed canonical datasets. A semantic layer where a human owns every metric definition. Lineage. Skills that encode how a careful analyst works. An eval harness watching all of it. Four layers, built and maintained by one of the best data teams in the world. I wrote about who gets to build that scaffolding when the post first landed.

The other vendors built the same conclusion into their products. Snowflake restricts its own MCP server to semantic views because raw schemas lack the meaning an analyst needs. Google wraps Gemini in a Knowledge Catalog and a Looker semantic layer and calls the bundle the product. The connector is the interface; the governed layer underneath is what they sell.

What a connector never builds

Accuracy aside, there is a category error in the pitch. A connector reads tables. Your data problem is that nobody is building the tables.

The connector will not land your sources in a warehouse. It will not model Raw into Clean into Ready. It runs when you open a chat and stops when you close it: ChatGPT sessions drop their state on disconnect and run nothing on a schedule. It watches nothing overnight. It re-learns your business every session, because chat memory is not a semantic layer.

And the plumbing itself is on you. Anthropic archived its reference Postgres MCP server, the one without query timeouts, row caps, or column blocklists, behind a plain warning: no security guarantees. It still gets about 312k installs a month, mostly from people wiring up exactly the setup in the pitch.

Where I land

OptimaFlo runs on these same models. You bring your own Claude, GPT, or Gemini key. I have no argument with the engine.

What we sell is the 74 points: the governed datasets, the semantic layer, the pipelines, the quality checks, and the eval discipline that Anthropic staffed a team to build, shipped as a product one data owner can run, on open Iceberg tables in your own cloud. A human approves the SQL before it touches your tables. The chat interface is in there too. It is the last step, and it is the easy one.

If your warehouse is already modeled and governed, a $20 chat seat on top of it is a great deal, and I say so in our side-by-side comparison. If nobody is building that warehouse, the seat has nothing to stand on.

The 21% is what the model brings. The rest is the job. Come see how we do it, or kick the tires in a 7-day pilot.