How Accurate Are AI Agents on Syndicated Data?
You trust an agent's answer when every query is checked against the provider's rulebook by fixed rules that always run the same way, not because the model sounds confident. Model fluency and model accuracy are different properties, and the published benchmarks put hard numbers on the gap.
What the benchmarks actually say
The most credible public measure of AI-writes-your-queries accuracy is BIRD, a benchmark from the text-to-SQL research field (the study of systems that turn a question into a database query), introduced at NeurIPS 2023 and built to look like the real world: 12,751 question-and-SQL pairs over 95 databases totaling 33.4 GB, spanning 37 professional domains, with deliberately dirty values and questions that require outside knowledge to answer correctly [1]. Its scoring metric, execution accuracy, is unforgiving: the generated query's results must match the correct query's results exactly. Earlier academic benchmarks used small, clean databases and flattered the models; BIRD's authors built theirs to expose the gap between lab results and production databases, which is exactly the gap a brand team needs to understand before pointing AI at a Circana extract [1].
Three numbers from it matter to a brand team. The BIRD paper reports human experts (data engineers and database students) at 92.96% execution accuracy on the test set [1]. At launch, the paper put the best available model, ChatGPT, at 40.08% [1]. And after two and a half years of the entire field optimizing against this exact target, the best purpose-built systems on the public leaderboard sit around 75-82% as of July 2026, with the top entry at 81.95% and plain GPT-4 at 51.75% even with the benchmark's curated knowledge hints [2].
Read that trajectory honestly. The progress is real: from 40% to 82% is remarkable. But the best system on earth, tuned for this exact benchmark, still gets roughly 1 query in 5 wrong, and a human expert outperforms it by 11 points. Now remove the benchmark's training wheels: your syndicated extract was not in anyone's training data, your measures carry rules about how they may be added up that the raw file does not spell out (the aggregation trap chapter walks through them), and nobody curated knowledge hints for your category. Raw question-to-query AI on syndicated data carries an error rate nobody has measured on your extract, and everything about the setup says it is worse than the leaderboard's.
Why "sounds confident" is not accuracy
The failure mode that matters is not the query that errors out. It is the query that runs.
A wrong SQL query on syndicated data almost never returns something visibly absurd. It returns a velocity that is 20% off because the period filter caught 5 weeks instead of 4, or a share number computed against the wrong category total, or a "national" distribution figure produced by averaging market-level % ACV. Each arrives formatted, charted, and narrated with the same fluency as a correct answer. The model's confidence stays constant while its accuracy varies.
This is why "we tested it and the answers looked right" is a weak evaluation. Looking right is the one thing wrong answers on syndicated data reliably do. The 8-11 point gap between the best systems and human experts on BIRD is made of exactly these silent failures.
The stakes are also asymmetric. A brand's syndicated numbers get quoted in buyer meetings, category reviews, board decks, and trade negotiations. One confidently wrong share number in front of a buyer costs far more than one answer; it costs the buyer's trust in every number you bring afterward, and that trust is most of what a data-backed sales story is for. An 80% accurate oracle is impressive technology and a commercial liability, because you cannot tell which fifth of its statements will detonate.
If you want to test a tool yourself, skip the demo script and run last period's real questions through it: the ones your team already answered by hand, where you know the right numbers. Then ask to see the query behind each answer. The percentage that match is your accuracy figure for that tool, and the vendor's reaction to the request is your governance figure.
What verification looks like
The engineering answer is to stop treating accuracy as a property of the model and start treating it as a property of the system. Sous is built on that premise, and the checks are concrete.
Every query is validated against the provider's rulebook before the answer ships: is the aggregation legal for this measure, is the period filter aligned to the provider's calendar, is the product level consistent through the whole calculation. The rulebooks are per-provider (SPINS, Circana, and Nielsen each get their own), because the legal math differs across them. These checks are fixed rules that run the same way every time. They do not depend on the model having a good day.
Just as important, every number stays traceable. The queries live inside the workbook next to the charts and narrative they produced, so a human can open any figure and audit the math behind it.
This architecture concedes that the model, alone, is not accurate enough for unsupervised syndicated math, which the BIRD numbers make hard to dispute. It keeps the model where models are strong (planning the analysis, drafting the queries, writing the narrative) and runs the cheap fixed-rule check on every single query. The combination gets you the model's flexibility with the rulebook's reliability, which neither has alone.
The practical standard for the category has three parts:
- The model plans and drafts.
- Fixed rules verify every query before the answer ships.
- A human can always inspect the math behind any number.
Systems missing the second and third parts are a different product that happens to share the chat box.
An honest word on the hype
Balance requires saying the uncomfortable part. Gartner forecasts that over 40% of agentic AI projects will be canceled by the end of 2027, citing escalating costs, unclear business value, and inadequate risk controls, and warns explicitly about "agent washing," vendors rebranding existing chatbots and automation as agents. By Gartner's estimate, only about 130 of the thousands of vendors claiming agentic capabilities are real [3].
The forecast argues for discipline, not against agents. The projects that will die are the ones that shipped confidence without verification. For a checklist you can bring to a demo, see "what should CPG teams look for in an analytics agent?"