Why Your AI Gives Different Answers to the Same Data Question

Ask an AI assistant the same question about your company data twice and you can get two different numbers. Most explanations blame the model – temperature, randomness, prompt wording. On your own data that is rarely the real cause. The variance comes from an ambiguous schema that forces the model to invent a join, guess a grain, and pick one of several competing definitions of "revenue" on every run.

Here is the version of this that happens in real companies.

A founder types "what was our revenue last month?" into an AI assistant wired to the company warehouse. It answers $412,000. Ten minutes later, in a different thread, they ask "how much revenue did we do in July?" and get $389,000.

Same data. Same month. Two answers, both delivered with total confidence, neither carrying a footnote explaining what it counted.

The instinct is to conclude the AI is unreliable and go back to asking the analyst. That instinct is half right – the answer was unreliable – but the diagnosis is wrong, and the wrong diagnosis leads to the wrong fix.

The explanation you will find everywhere

Search this question and you will get a consistent answer about how language models work. It is accurate as far as it goes.

Sampling and temperature

Language models predict the next token as a probability distribution and then sample from it. That sampling introduces randomness by design – it is what makes the output feel natural rather than robotic. A `temperature` setting controls how much. Set it to zero and the model always takes the most likely token.

Non-determinism runs deeper than temperature

Even at temperature zero, identical prompts can produce different outputs. Thinking Machines Lab traced the dominant cause to a lack of batch invariance in inference kernels – the arithmetic changes subtly depending on how requests are batched together on the server. Their work shows that with batch-invariant kernels, 1,000 identical runs produce 1,000 identical outputs.

So non-determinism is a real engineering problem, and it is a solvable one.

Why none of this explains your revenue number

Here is the thing that should bother you: a $23,000 gap between two revenue answers is not a rounding artifact of token sampling. Sampling variance changes phrasing. It does not move a number by six percent.

Something else is happening, and it is happening in the step everyone skips over – between your question and the SQL that runs against your warehouse.

What actually goes wrong when AI queries your data

When a language model translates a business question into SQL, it has to make a series of decisions that nobody told it how to make. Each one is a place where two runs can legitimately diverge.

It invents joins

Your database or data warehouse has an `orders` table and a `customers` table. Are they joined on `customer_id`, or on `user_id`, or through an intermediate `accounts` table? If the relationship is not declared anywhere the model can read, it infers one from column names.

Sometimes that inference is right. Sometimes it produces a join that silently duplicates rows and inflates the total. The query runs, returns a number, and nothing indicates a guess was made.

It guesses the table grain

Does one row in your `subscriptions` table represent a subscription, a billing period, or a plan change? Aggregating the wrong grain double-counts revenue in a way that looks entirely plausible.

It applies filters nobody asked for

Should test transactions be excluded? Internal transfers? Refunded orders? Your analyst knows the answer because they have been burned before. The model has no such history, so on one run it might filter `status = 'completed'` and on the next include everything.

It picks one of your definitions of revenue

This is the largest source of variance and the least technical. If marketing counts gross bookings, finance counts net receipts, and the CRM counts closed-won value, then "revenue" has three legitimate answers already sitting in your warehouse. The model picks one. There is no reason it picks the same one twice.

The two queries, side by side

Strip away the chat interface and the $412,000 answer and the $389,000 answer are two SQL statements that differ by a handful of tokens. Here is the shape of the tables the model was handed:

1orders(order_id, account_id, created_at, status, amount_gross)
2refunds(refund_id, order_id, refunded_at, amount)
3accounts(account_id, user_id, is_internal, plan)

Nothing in that schema says whether a refund belongs to the month of the original order or the month it was issued, that accounts flagged `is_internal` are excluded from company revenue, or that `status` contains a value called `authorized` for charges that never settled. The model decides all three, fresh, on every run.

The first run wrote the simple thing:

1SELECT SUM(amount_gross) FROM orders
2WHERE created_at >= '2026-07-01' AND created_at < '2026-08-01';

The second run was more careful:

1SELECT SUM(o.amount_gross) - COALESCE(SUM(r.amount), 0)
2FROM orders o
3LEFT JOIN refunds r ON r.order_id = o.order_id
4JOIN accounts a ON a.account_id = o.account_id
5WHERE o.created_at >= '2026-07-01' AND o.created_at < '2026-08-01'
6  AND o.status = 'completed' AND a.is_internal = FALSE;

Both statements are valid. Both executed without error. Both returned a tidy figure with a dollar sign in front of it. The gap between them is refunds issued in July plus a handful of internal test orders – business logic, not arithmetic.

And the second query, the careful one, carries a defect of its own. The `LEFT JOIN` to `refunds` fans out any order with two partial refunds into two rows, so `SUM(o.amount_gross)` counts that order's gross amount twice. A more sophisticated query can be more wrong than a naive one, and it looks more trustworthy while it happens.

The AI is not hallucinating here. It is disambiguating – repeatedly, invisibly, and without a rule to follow.

The benchmark almost nobody cites

There is published evidence for exactly how badly this goes, and it is startling once you see the two numbers side by side.

On Spider, the Yale text-to-SQL benchmark built on clean academic schemas, top systems reach around 91% execution accuracy. That is the number that makes AI-on-your-data look solved.

Then there is BEAVER, a benchmark built from real enterprise data warehouses rather than tidy teaching databases. On BEAVER, state-of-the-art systems reach 10.8% accuracy. With oracle hints supplied on subtasks, it rises to 30.1%.

Roughly 91% on clean schemas. Roughly 11% on real corporate warehouses. The gap is not the model. The gap is the schema.

BIRD, which sits between the two in realism, tells a consistent story: human performance is 92.96% execution accuracy while the best model on the leaderboard reaches 81.95%.

The uncomfortable implication: the demos you have seen were probably run against something closer to Spider than to your warehouse.

Determinism is not correctness

This is the part that matters most, and it is where most advice on this topic quietly fails.

Suppose you fix everything in the first section. Temperature zero. Batch-invariant kernels. Fully reproducible inference. Now you ask the same question twice and get the same answer both times.

You have made the answer consistent. You have not made it right.

A stable wrong number is the one nobody checks

Inconsistency is annoying, but it is also a smoke alarm. The $23,000 gap is the reason anyone looked under the hood at all. Two different answers force a conversation about what "revenue" means; one answer, repeated, forces nothing.

Make that system deterministic and the alarm goes silent while the fire keeps burning. The fanned-out `LEFT JOIN` over-counts by the same amount every time, so the number stops moving and starts looking like a fact – board deck in April, investor update in May, compensation plan in June. Because each figure agrees with the last, every check you would think to run confirms it.

Reproducibility launders a guess into a fact. By the time someone notices, the error is not in one slide. It is in the trend line, and the trend line is what people decided on.

Auditing a number back to its definition

The property you want is not "the same answer twice." It is that every figure resolves to a named metric (`revenue_net`, not the word "revenue"), a stored SQL definition, and an author with a version history. That is the difference between a number that arrived and a number that was issued.

Here is the test. Take the revenue figure in your last board deck and walk it backwards.

1.Which named metric is it? If the answer is "the revenue number," stop – there is no definition to audit, only a query somebody ran.

2.Show the SQL. Not the dashboard, not the export: the join path, the grain, the `WHERE` clause.

3.Read the filters out loud. Internal accounts? Refunds? Authorized-but-unsettled charges? Each one you cannot answer is a place two runs can diverge.

4.Ask who last changed it and why. With no change history, the definition drifts and nobody notices that last quarter and this quarter were computed differently.

A system that survives those four steps can still be wrong – but it can be corrected, once, at the source. A system that fails them cannot be corrected at all, only re-litigated.

Consistency is not the goal. Traceability is – being able to say which definition produced this number and who wrote it.

What actually fixes it

If the problem is ambiguity in the schema, the fix is removing the ambiguity – not swapping models.

The evidence for this is measurable

A 2026 paired benchmark across three frontier models tested 100 natural-language questions with and without a semantic document describing the data. Without it: 45.5–50.5% accuracy. With it: 67.7–68.7%.

A 4 KB markdown document describing what the tables mean improved accuracy by 17 to 23 percentage points. The finding that should reframe your roadmap: with the semantic document in place, which model you used stopped mattering in a statistically meaningful way.

You are probably not one model upgrade away from trustworthy answers. You are one data model away.

The analyst writes the definition once

A governed data model is where the ambiguity gets resolved in advance. The join path is declared. The grain is stated. The filters are written down. Revenue means one thing, in SQL, authored by a person who is accountable for it.

The AI then reads from that model rather than composing its own interpretation of your warehouse. It is not guessing at joins because the joins are given. Every number traces back to human-written SQL, which means someone can be asked to defend it.

What this does not fix

Worth being straight about the boundary. A data model does not make the AI's prose correct, only its numbers. It does not eliminate the analyst – someone still writes and maintains the definitions. And it does not help with questions the model does not cover; those still require new work.

What it does is collapse the answer space for the questions it does cover, from "several defensible interpretations" to one.

The objections worth taking seriously

Three pushbacks come up every time this argument is made. Each is cheaper than building a data model, which is exactly why each deserves a straight answer.

"Can't I just prompt better?"

Partly, and it is worth doing – telling the assistant "revenue means completed orders, net of refunds, excluding internal accounts" removes three of the ambiguities above. But that instruction lives in one chat thread. Your CFO's thread does not have it, nor does the scheduled report, the Slack query, or you in three weeks. A prompt is a definition scoped to one conversation. Paste the join paths in every time and you have written a data model anyway, in the least durable place available.

"Won't the next model fix this?"

This is what the paired benchmark above was built to test, and the result points the other way: with the semantic document in place, the choice of frontier model stopped mattering. A better model reasons better about the information it has; it cannot recover information nobody wrote down. Whether refunds belong to the order month or the refund month is not latent in your schema waiting to be inferred – it is a decision your finance team made, or failed to make.

"What about RAG over my schema?"

Retrieval over table and column names helps the model find the right tables. It does not tell the model what they mean: retrieving `orders.status` gives you the column, not the fact that `authorized` means the card was held and never captured. Retrieval over your documentation is a real improvement – but somebody had to write that documentation. RAG is a delivery mechanism for definitions, not a substitute for having them. Point it at a wiki page last edited two years ago and it delivers a stale definition, with citations attached.

How you know it is working

One signal you can check next week, one that only surfaces a quarter in.

The early signal

Ask the same question three ways – "revenue last month," "how much did we bill in July," "July top line" – in three separate sessions. You want the same number three times, each answer naming the metric it used. The stronger test is disagreement: "that is not what I get" should become a two-minute comparison of two named definitions, not a two-day reconciliation. The argument moves from the number to the definition, the only place it can be settled.

What failure looks like at month three

Failure is quiet erosion, not a crash, and it has a recognisable shape:

•A question the model did not cover got a one-off query, which quietly became a recurring report

•A definition changed and old figures were never restated, so the trend line has an unlabelled seam

•Two metrics now exist for one concept – `revenue` and `revenue_v2` – and nobody knows which the board deck uses

•The analyst who authored the definitions left, the SQL is still there, and nobody feels entitled to change it

All four start the same way: a definition was needed, and answering the question was faster than writing it down. The health check is not whether the numbers agree. It is whether the count of ungoverned queries is going up or down.

Where to start

You do not need a governance program. Start with the three metrics your leadership team argues about most – usually revenue, customer acquisition cost, and some version of active customer – and get one written definition for each, in SQL, in a place every tool queries.

That is the entire intervention. Everything else is elaboration.

With OWOX Data Marts, the model lives in your own warehouse and the definitions are authored by your analyst. AI assistants query the governed model instead of raw tables, so the answer to "what was our revenue last month?" is the same on Tuesday as it was on Monday – and, more importantly, it is the same as the number in the board deck.

Ask the question twice. Getting the same answer is the low bar. Getting an answer somebody can defend is the actual one.