---
title: "How to review AI-generated SQL: checks tested on 36 runs"
canonical: "https://www.owox.com/blog/articles/review-ai-generated-sql"
updated: "2026-10-14"
---

# How to review AI-generated SQL: the checks, in order, tested on ChatGPT and Claude

Someone hands you SQL an AI wrote and asks you to confirm the number. We ran six business questions on ChatGPT (Free plan) and on Claude, 24 plain runs plus 12 in which Claude also wrote a check query, and kept every wrong answer. In the 24 plain runs every query passed the dry run and executed, the textbook traps did not fire, and the five wrong answers came back empty, NULL or zero with no error. This guide gives the review order that follows, three checks as copyable SQL with their real output, and the limits of the test.

[AI in Analytics](/blog/topics/ai) · Updated October 14, 2026 · 15 min read

![How to review AI-generated SQL: the checks, in order, tested on ChatGPT and Claude](https://cdn.owox.ai/www/content-marketing-article/review-ai-generated-sql/thumbnail/1.png/public)

Someone hands you a query an AI assistant wrote, or a number it produced, and asks you to confirm it. **You are the person whose name ends up on that number.** This page is the order in which to check it, built on what 36 test runs of ChatGPT and Claude actually got wrong.

**Can your team ask the AI directly?** Our advice, from 24 plain runs on one demo warehouse: yes to draft a query, no to passing its number on because the query ran. Five of the 24 answers were wrong (a count on one demo warehouse, not a rate), no error flagged any of them, and the checks below caught all five, within the limits stated under the short answer. So whoever owns the definitions of your data, usually the analyst, checks each figure against a total from the raw table before it leaves the team.

Both models passed a BigQuery dry run (a check that the query is valid, made without running it) and executed every time, the textbook traps fired in none of the runs, and **nothing errored.** The five wrong answers came back empty, zero or NULL. Four came from the casing of a status value, `'completed'` where the data says `'Completed'`; the fifth from a value ChatGPT invented, `'delivered'`, that is not in the data at all.

We ran the same six questions on ChatGPT (Free plan) and on Claude on 14 October 2026. Every result carries its model label, and none of it transfers to other models, plans or warehouses. If you ask an assistant for SQL for your own report and cannot judge a query by reading it, [how to ask an assistant for SQL](/blog/articles/how-to-build-sql-queries-with-chatgpt) is the shorter path. This page is for the person who reads the query.

## The short answer

Review in this order. The count after a step is how many of the five wrong answers in our 24 plain runs it caught.

1.  **Do not count the dry run, or “it ran”, as evidence.** All five wrong answers passed both.
2.  Reconcile one total (C1): compare the answer’s total with a total computed another way from the raw table. It caught 5 of 5.
3.  Look up every typed-in value the query filters on (a literal, such as a status name) in the data. The schema names a column; it does not list its values. We did not score this lookup as a check, but it exposes the cause.
4.  Count rows after each filter and join (C3): how many rows survive each step. It caught 5 of 5, as a zero-row signal; none of our runs multiplied rows.
5.  Then read the query: grain, the definition of the metric, the date window.

Comparing the row count with the number of things you expect (C2), such as 7 rows for 7 sources, caught 0 of 5, so it is not a step.

**Read “5 of 5” as one failure shown five times, not five independent catches.** All five share one cause, a value the model could not see, and all five came back empty, NULL or zero, which a total exposes at once. No subtle wrong figure, a plausible number that is a little off, occurred in 24 runs, so these checks are untested on that kind. And a total can match while its split is wrong: right revenue, wrong channels. **Run C1 first. If the total moves, read the joins and the filters.**

## What we ran

We used the demo shop warehouse in BigQuery, `owox-demo.ecommerce`, and gave each model only the schema as text: table names, column names and types for five tables, no descriptions and no values. The prompt was the schema, one question asked the way a business person asks it, and “Write me the SQL.”, identical for both models and with no hint about traps. **The models never touched BigQuery.** We ran each returned query read-only, dry run first, and compared it with a reference query we wrote and froze before the first run: same rows, same values at the stated decimals.

The six questions were about Q3 2026: click-through rate by campaign, cost per click by source, ad cost per session by source, return on ad spend (ROAS) by source (net revenue of completed orders, attributed by the session’s source, over ad spend), discounts given away, and the share of order lines sold below list price. Each could trip a classic trap. **Each model got every question twice**, 24 plain runs in all.

|  | Claude (12 runs) | ChatGPT, Free plan (12 runs) |
| --- | --- | --- |
| Passed the dry run | 12 | 12 |
| Executed | 12 | 12 |
| Matched the reference | 10 | 9, or 6 with exact units |
| Loud error | 0 | 0 |
| Wrong with no error (empty, NULL or all zero) | 2 | 3 |
| Stated its assumptions | 12 | 9 |

Counts, not rates: two runs per question cannot give a rate. ChatGPT answered in percent in three runs where the reference is a fraction. We decided to count those as correct after the first ChatGPT reply came in, not before the runs; without that rule it matched 6 of 12.

## The Turning Point

OWOX Data Marts

See your first report built in real time. _15 minutes._

1.  Connect your data warehouse
2.  Pick your metrics
3.  Get a live Google Sheets report

In the time it takes to write a ticket. Then imagine never writing that ticket again.

[Book a Demo](/book-a-call)

We'll use your actual use case

## A passed dry run proved nothing

A BigQuery dry run checks that a query is valid and reports the bytes it would process; it does not run it. A filter that matches no rows is valid, so it passes. **All five wrong answers passed the dry run and executed.**

Here is ChatGPT’s ROAS query from the first of its two runs on that question. The dry run came back valid with 7,578,510 bytes to process. The query returned this:

| source | net\_revenue | ad\_spend | roas |
| --- | --- | --- | --- |
| facebook | 0.0 | 15415.55 | 0.0 |
| tiktok | 0.0 | 10050.89 | 0.0 |
| bing | 0.0 | 8877.82 | 0.0 |
| google | 0.0 | 8649.6 | 0.0 |
| linkedin | 0.0 | 4111.81 | 0.0 |
| reddit | 0.0 | 1537.73 | 0.0 |
| twitter | 0.0 | 419.14 | 0.0 |

Seven sources, spend that is right, and a return of zero on every one. **It looks like an answer.** The query wrapped its revenue in `COALESCE(..., 0)`, which turned a missing revenue into a clean 0.0. Claude’s queries returned NULL in those two columns, which is easier to notice and just as quiet.

## The traps you were told to hunt did not fire

Public checklists such as [Basedash’s](https://www.basedash.com/blog/how-to-review-ai-generated-sql-a-checklist-for-analysts-and-operators) and [this walk-through on DEV](https://dev.to/michaelnocito/how-to-review-ai-generated-sql-before-you-trust-the-number-19ek) include join fan-out and wrong denominators. We built the questions so those traps could fire: ad spend is stored one row per day per campaign, and the click-through column is stored per row. They fired in **0 of 24 plain runs**.

No run averaged per-row ratios; every ratio was a sum over a sum, and both models said so. ChatGPT’s reply to the cost-per-click question: “Aggregates spend and clicks before calculating CPC, rather than averaging individual CPC values”. The two questions that join ad spend to sessions (cost per session and ROAS, 8 of the 24 runs) aggregated both sides before joining in every run. The other four read one table, or join at order-line level and aggregate afterwards. That is a finding about these two models on this schema, not a promise that the traps never fire. A review aimed only at them would have flagged nothing here, and all five wrong answers would have gone through. For what hallucinated SQL looks like and why some teams remove the step, see [AI analytics hallucination: why it happens and the SQL fix](/blog/articles/ai-analytics-hallucination-deterministic-sql).

## The failure that did happen: a value the schema text did not contain

The schema text says `orders.status` is a string. It does not say what the strings are. In Q3 2026 the data holds 1,652 `Completed` orders, 182 `Cancelled` and 108 `Returned`. The question said “completed orders”, so each model had to guess the spelling.

On ROAS, all four runs (two Claude, two ChatGPT) wrote `status = 'completed'`, which matches nothing: NULL for Claude, 0.0 for ChatGPT. On the discounts question, ChatGPT’s second run filtered on `'delivered'`, a value that does not exist in the data. Its result was NULL, and its reply did not say the value was a guess. **Five wrong answers, no errors.**

The warnings were in the text of every ROAS reply, below the code:

*   Claude, first run: “I can’t see the actual status values, so run SELECT DISTINCT status FROM orders and adjust if it’s spelled differently, for example ‘Completed’ or ‘complete’.”
*   ChatGPT, second run: “This assumes orders.session\_id identifies the attributing session and that status = ‘completed’ is the exact status value used in your data.”
*   ChatGPT’s discounts reply stated “includes only delivered orders placed in Q3 2026” and offered to remove the filter; it never said the word was invented.

All four ROAS replies called the literal an assumption, and **all four wrote it anyway**. A reviewer who reads the code first sees a valid filter. The same spelling cost three of ten misses in our 11 October benchmark, described in [text-to-SQL accuracy: what a benchmark score tells you about your own warehouse](/blog/articles/text-to-sql-accuracy-benchmarks-production).

Neither model saw the data, so neither could look the value up. This is what a reader who pastes a schema gets, not a statement about either model in general. A loud failure is noticed at once; this kind costs a number in a deck.

## Three cheap checks, with real output

We ran the checks on all five wrong answers with no reference answer in hand, only the base tables. Each is a query you run beside the one under review; C1, C2 and C3 are defined in the short answer above. We did not time the checks, so we give no duration.

### C1: the total reconciles

Compute the base total without the filter you are testing (net revenue is zero for cancelled and returned orders in this schema, so dropping the status filter is safe here). This is the base side for the ROAS question; it returns 849,177.41 of net revenue and 49,062.54 of ad spend for Q3.

```
SELECT
  (SELECT SUM(p.line_net_revenue)
     FROM `owox-demo.ecommerce.orders` AS o
     JOIN `owox-demo.ecommerce.purchases` AS p ON p.order_id = o.order_id
    WHERE o.order_date >= DATE '2026-07-01' AND o.order_date < DATE '2026-10-01') AS base_net_revenue,
  (SELECT SUM(spend)
     FROM `owox-demo.ecommerce.ad_performance`
    WHERE date >= DATE '2026-07-01' AND date < DATE '2026-10-01') AS base_ad_spend;
```

Then paste the query under review as a CTE named `answer` and put its total beside the base total. Here it is on ChatGPT’s discounts query, short enough to print whole:

```
WITH answer AS (
  SELECT SUM(GREATEST(p.price - pu.sale_price, 0) * pu.quantity) AS total_discounts
  FROM `owox-demo.ecommerce.purchases` AS pu
  JOIN `owox-demo.ecommerce.products` AS p ON pu.product_id = p.product_id
  JOIN `owox-demo.ecommerce.orders` AS o ON pu.order_id = o.order_id
  WHERE o.order_date >= DATE '2026-07-01' AND o.order_date < DATE '2026-10-01'
    AND o.status = 'delivered'
)
SELECT
  (SELECT total_discounts FROM answer) AS answer_total,
  (SELECT SUM(pu.quantity * (p.price - pu.sale_price))
     FROM `owox-demo.ecommerce.purchases` AS pu
     JOIN `owox-demo.ecommerce.products` AS p ON pu.product_id = p.product_id
     JOIN `owox-demo.ecommerce.orders` AS o ON pu.order_id = o.order_id
    WHERE o.order_date >= DATE '2026-07-01' AND o.order_date < DATE '2026-10-01') AS base_total;
```

It returns NULL against 52,188.49. **C1 caught all five.** Wrapped the same way, ChatGPT’s ROAS query gave net revenue of 0.0 against 849,177.41, while its ad spend matched at 49,062.54, which is why the spend column looked fine.

### The literal lookup

For every string in a WHERE clause, ask the data what the column holds:

```
SELECT status, COUNT(*) AS orders
FROM `owox-demo.ecommerce.orders`
WHERE order_date >= DATE '2026-07-01' AND order_date < DATE '2026-10-01'
GROUP BY status
ORDER BY orders DESC;
```

It returns Completed 1,652, Cancelled 182 and Returned 108: the cause of all five wrong answers, in three rows. We ran it after the fact and did not score it as a check.

### C3: rows after each filter and join

The step where rows drop to zero, or multiply, is the fault. This query hardcodes the literal we already knew was wrong. On a real review you do not know where to look, which is why the lookup comes first: list every string in the WHERE clauses, then add the query’s own filters and joins one at a time and count after each.

```
WITH q3 AS (
  SELECT * FROM `owox-demo.ecommerce.orders`
  WHERE order_date >= DATE '2026-07-01' AND order_date < DATE '2026-10-01'
)
SELECT '1 orders in Q3' AS step, COUNT(*) AS n FROM q3
UNION ALL
SELECT '2 after the status filter', COUNT(*) FROM q3 WHERE status = 'completed'
UNION ALL
SELECT '3 after the join to purchases', COUNT(*)
FROM q3 JOIN `owox-demo.ecommerce.purchases` AS p USING (order_id)
WHERE q3.status = 'completed'
UNION ALL
SELECT '4 after the join to sessions', COUNT(*)
FROM q3
JOIN `owox-demo.ecommerce.purchases` AS p USING (order_id)
JOIN `owox-demo.ecommerce.sessions` AS s ON s.session_id = q3.session_id
WHERE q3.status = 'completed'
ORDER BY step;
```

| Step | `'completed'`, as the models wrote it | `'Completed'`, as the data has it |
| --- | --- | --- |
| 1 orders in Q3 | 1,942 | 1,942 |
| 2 after the status filter | 0 | 1,652 |
| 3 after the join to purchases | 0 | 2,964 |
| 4 after the join to sessions | 0 | 2,964 |

The rise from 1,652 to 2,964 at step 3 is expected, because an order has several lines; what you look for is step 4 leaving 2,964 as 2,964. DEV’s walkthrough also puts this check first. What our runs add is that the step it exposed was a filter, not a join.

To see what it shows when rows do multiply, we wrote a deliberately naive query ourselves; no model wrote it. Joining ad spend to sessions on source and date turns the 1,500 ad rows of Q3 into 197,617 rows and a spend of 8,520,956.26 against a true 49,062.54. The row count gives it away before the total does.

### C2: the row count, which caught nothing

The ROAS answers had 7 rows, and 7 sources had ad spend in Q3. The row count agreed on all five wrong answers: the ROAS queries took their rows from the spend side, which was right, and the discounts query is an aggregate without a GROUP BY, so it returns one row whatever its filter matched. C2 checks grain, not filters.

## Asking the assistant for a check query

Another habit is to ask the assistant to check its own work. We tested that on Claude only, not on ChatGPT. Twelve fresh runs got the same schema and questions plus: “Write me the SQL, and also give me a second query that checks your figure against an independent total, so I can tell if yours is wrong.”

Counted strictly, the main queries came out at 8 correct, 1 loud error and 2 empty, the same two ROAS misses with the same lower-case literal. One more discounts run returned three rows, one per status, with no total; they sum to the right figure, but the reader has to add them. The plain Claude arm had 10 correct; the new loud error was a BigQuery message, “Aggregations of aggregations are not allowed”, on one discounts run. The check queries did worse than the questions:

*   **8 of 12 checks ran. 4 of 12 failed with a BigQuery error:** three times “Correlated subqueries that reference other tables are not supported unless they can be de-correlated”, once the aggregation error.
*   **Of the two silent failures, the check caught one.** On the first ROAS run the check printed MISMATCH, but only because a comparison against NULL falls to the else branch; both sides of that check reused the same wrong filter, so it could not say why. On the second ROAS run the check itself failed with the correlated-subquery error.
*   **No false alarms:** the 7 checks that ran against correct answers all agreed.

One reply stated its own limit: “If status = ‘completed’ matches zero rows, both queries return NULL revenue, so confirm the status values first.” The check used the same wrong literal anyway. **A check that reuses the filter under test confirms that filter by construction.** Treat a check query as a draft, run it like any query, and write the independent total yourself from the base table. One thing would have worked: the second ROAS reply ended with a status-listing query that, run as written, returns the three real values.

## ChatGPT and Claude, side by side

Matches with the reference, out of two runs per question:

| Question | Claude | ChatGPT, Free plan |
| --- | --- | --- |
| Click-through rate by campaign | 2 | 2 (both in percent) |
| Cost per click by source | 2 | 2 |
| Ad cost per session by source | 2 | 2 |
| Return on ad spend by source | 0 | 0 |
| Discounts given away | 2 | 1 |
| Share of order lines below list price | 2 | 2 (one run in percent) |

**Both models failed the same question the same way**, with the same lower-case literal. They differed in three places: ChatGPT had one more failure, the invented `'delivered'` filter nobody asked for; its `COALESCE` turned the missing revenue into 0.0 where Claude returned NULL; and it stated its assumptions in 9 of 12 replies against Claude’s 12 of 12. **A gap of one run in 12 does not say which model is better**, and we make no such claim.

The limits are specific:

*   One warehouse, one demo shop. We wrote the questions and the references and froze the references before the first run.
*   Two runs per question per model, so counts only.
*   ChatGPT ran on the Free plan as the interface showed it, in temporary chats. The interface did not display a model label, and the chat’s personalization setting read “Personalized” and could not be switched, so the account’s memory may have been available.
*   Claude ran as fresh sub-agents given only the schema text. The run files did not record a model version, so this page names none.
*   Six of the twelve plain Claude replies were saved by us from the sub-agents’ returned text, because the sub-agents did not write their files; the text is unchanged. In the check arm one reply was recovered from the workflow journal and one run was repeated after an API safeguard error.
*   Neither model saw the data. With access to it, a model could look a status value up.

## When to stop reviewing and move the definition

Every wrong answer here hinged on a definition: what counts as completed. Review catches it each time someone asks, and then the next question asks again.

[OWOX Data Marts](/product) is our product: an analyst defines each Data Mart, a dataset written as SQL with a description and its joins to other Data Marts, in the company’s own warehouse, as the [data model](/product/data-modeling). An [AI assistant](/product/ai-assistants) connected through the [OWOX MCP server](/features/mcp) picks fields from published Data Marts instead of writing SQL, and each query is kept in Run History with its executed SQL. That lowers the risk of a wrong figure; **it is not a guarantee.** In our 11 October test the published-Data-Mart setup led with a wrong revenue figure once and declined once in 22 runs, as the benchmark article above lays out.

## Your New Normal

Turn your data into decisions.

Governed data marts give you the clean foundation ML needs to actually work.

*   No AI hallucinations
*   Analyst-governed definitions
*   Every number traces to SQL

[Get started free](https://www.owox.com/app-signup)

## FAQ

## Frequently Asked Questions

How do I check SQL that ChatGPT wrote?

Do not count a passed dry run, or the fact that the query ran, as evidence. In our 24 plain runs on ChatGPT and Claude, all five wrong answers passed the dry run and executed without an error.

Check in this order. First, reconcile one total: compute the same total another way from the raw table, without the filter under test, and put it beside the query's answer. It caught all five wrong answers. Second, look up every typed-in value the query filters on, such as a status name, in the data, because a schema names a column but does not list its values. Third, count the rows left after each filter and join; the step where rows drop to zero is where the query lost its data. Then read the query: grain, the definition of the metric and the date window.

The counts come from one demo warehouse, six questions, ChatGPT (Free plan) and Claude, and none of it transfers to other models, plans or warehouses. All five wrong answers shared one cause, so read the five catches as one failure shown five times.

Can ChatGPT check its own SQL?

We did not test ChatGPT on this. We tested the habit on Claude, in 12 runs where it was asked for its SQL plus a second query that checks its figure against an independent total.

Eight of the 12 check queries ran; 4 failed with a BigQuery error. Of the two silent wrong answers, the check flagged one, and only because a comparison against NULL falls to the else branch; on the other the check itself failed with an error. In one reply the check reused the same wrong status value as the main query, and a check that reuses the filter under test confirms that filter by construction.

Treat an assistant-written check as a draft: run it like any query, and write the independent total yourself from the base table. This is one model, one warehouse and 12 runs.

Why did the SQL run without errors but return empty, zero or NULL?

A valid query can match no rows. A BigQuery dry run checks that a query is valid, not what it returns, and a filter that matches nothing is valid, so it passes and executes.

In our test all five wrong answers had one cause: a value the model could not see. The schema text says a status column is a string but not what the strings are. Four runs filtered on 'completed' where the data says 'Completed', which gave NULL for Claude and 0.0 for ChatGPT, whose query wrapped revenue in COALESCE(..., 0). The fifth filtered on 'delivered', a value that is not in the data.

To find it, list the real values of the column (group the orders by status and count them) and count the rows left after each filter and join. The step where the count drops to zero is the fault. Neither model saw the data in our test, so this is what a reader who pastes a schema gets.

Which is better at SQL, ChatGPT or Claude?

Our test does not rank them. On six business questions on one demo warehouse, each model got 12 plain runs. Claude matched the reference in 10 and ChatGPT (Free plan) in 9, or in 6 if units are counted exactly, because three ChatGPT answers were in percent where the reference is a fraction.

Both failed the return-on-ad-spend question in all four runs, with the same lower-case status value. ChatGPT had one more failure, an invented 'delivered' filter that nobody asked for, and stated its assumptions in 9 of 12 replies against Claude's 12 of 12.

A gap of one run in 12 does not say which model is better. These are counts, not rates: two runs per question, questions we wrote ourselves, and no Claude model version recorded.

Does a passed BigQuery dry run mean the SQL is right?

No. A BigQuery dry run checks that a query is valid and reports the bytes it would process; it does not run the query. A filter that matches no rows is valid, so it passes.

In our 24 plain runs every query passed the dry run and executed, including all five wrong answers. One ChatGPT query came back valid with 7,578,510 bytes to process and then returned a return on ad spend of 0.0 for all seven sources.

The dry run tells you the query is valid and what it would scan. To know the number is right, reconcile a total against the raw table and count rows after each filter and join.

## Topics

[AI in Analytics](/blog/topics/ai)

## Related articles

[AI in Analytics · AI analytics agents replace your backlog, not your analyst · July 2, 2026](/blog/articles/ai-analytics-agent-replace-backlog)

[AI in Analytics · AI hallucinating SQL: what it looks like and how to catch it · October 14, 2026](/blog/articles/ai-analytics-hallucination-deterministic-sql)

[AI in Analytics · Trace an AI answer to its SQL: from number to source table · October 14, 2026](/blog/articles/ai-analytics-hallucinations-sql-traceability)

[See all articles →](/blog/articles)

## Book a call

You are booked. The invitation is on its way to , with the calendar invite and the meeting link.

[Reschedule](#)

[Cancel](#)

## Links

Pages this page links to, on this site and on docs.owox.com. Where the page has a Markdown twin, its address follows the link.

- [AI in Analytics](https://www.owox.com/blog/topics/ai) — /blog/topics/ai.md
- [how to ask an assistant for SQL](https://www.owox.com/blog/articles/how-to-build-sql-queries-with-chatgpt) — /blog/articles/how-to-build-sql-queries-with-chatgpt.md
- [Book a Demo](https://www.owox.com/book-a-call) — /book-a-call.md
- [AI analytics hallucination: why it happens and the SQL fix](https://www.owox.com/blog/articles/ai-analytics-hallucination-deterministic-sql) — /blog/articles/ai-analytics-hallucination-deterministic-sql.md
- [text-to-SQL accuracy: what a benchmark score tells you about your own warehouse](https://www.owox.com/blog/articles/text-to-sql-accuracy-benchmarks-production) — /blog/articles/text-to-sql-accuracy-benchmarks-production.md
- [OWOX Data Marts](https://www.owox.com/product) — /product.md
- [data model](https://www.owox.com/product/data-modeling) — /product/data-modeling.md
- [AI assistant](https://www.owox.com/product/ai-assistants) — /product/ai-assistants.md
- [OWOX MCP server](https://www.owox.com/features/mcp) — /features/mcp.md
- [Get started free](https://www.owox.com/app-signup)
- [AI in Analytics · AI analytics agents replace your backlog, not your analyst · July 2, 2026](https://www.owox.com/blog/articles/ai-analytics-agent-replace-backlog) — /blog/articles/ai-analytics-agent-replace-backlog.md
- [AI in Analytics · Trace an AI answer to its SQL: from number to source table · October 14, 2026](https://www.owox.com/blog/articles/ai-analytics-hallucinations-sql-traceability) — /blog/articles/ai-analytics-hallucinations-sql-traceability.md
- [See all articles →](https://www.owox.com/blog/articles) — /blog/articles.md