Talking to company data – why text-to-SQL returns answers that look right and are wrong

The gap between a demonstration and production has hard support in the benchmarks. On Spider 1.0 – an academic set with readable schemas – agents reach around 91%. The same agent on Spider 2.0, a set built from real enterprise workflows, drops to 21,3%. That is not a difference in model quality; it is a difference in what the schema looks like.

A real warehouse in a mid-sized company holds hundreds of tables, columns named with abbreviations from the nineties, technical columns beside business ones, fields populated differently depending on who entered the record, and several generations of solutions nobody removed because „something still uses it”. The model receives all of it at once and has to guess which field corresponds to the concept the person is asking about.

Then there is the join problem. „How much did we sell last quarter to premium-segment customers” takes one join in a well-modelled warehouse. In a warehouse that grew organically it takes six – and every one of them is a place to lose or duplicate rows. The dbt benchmark showed this directly: some questions were out of reach for one approach precisely because they required too many hops between tables.

The conclusion is inconvenient for tool vendors but simple for planning: the effectiveness of conversational analytics in your company depends mainly on the state of your data model, not on which engine you choose.

According to Omni's analysis, 81,2% of text-to-SQL errors arise at the schema or semantic level rather than in syntax. The distinction matters operationally: a syntax error ends in a message you cannot miss. A semantic error ends in a number.

The most common example: „what was our revenue in June”. A warehouse often holds three answers – the net value of invoices issued in June, the value of orders fulfilled in June, and the value recognized in the accounts in June. Each is correct from some department's perspective and each yields a different figure. The model picks the first field that looks right and reports the result without qualification.

The second family of errors concerns filters nobody states out loud because they are obvious to people inside the company: that cancelled documents do not count, that the subsidiary is reported separately, that corrections belong to the month of the original invoice. Those rules live in four people's heads and in the code of a few reports, but not in the schema – so the model will not apply them.

The third family is units and currencies. A price column with no indication of whether it is net or gross, in what currency, converted at what rate, is simply a number to the model. The answer will contain a total that looks exactly as it should – and will be wrong.

A semantic layer is a set of definitions sitting between the database and the person asking: a list of metrics with an unambiguous calculation, the dimensions they may be sliced by, the relationships between entities, and descriptions in the language the company actually speaks. The model then does not write a query from scratch – it selects from a closed set of concepts.

The effect shows in the numbers. In the dbt Labs benchmark from 2026, grounding in a semantic layer raised accuracy from 90,0% to 98,2% for Claude Sonnet 4.6 and from 84,1% to 100% for GPT-5.3-Codex. It is worth knowing the scale of that measurement: eleven questions run twenty times each on a single insurance dataset. That is a strong directional signal, not proof of universal effectiveness – which is precisely why it is worth measuring the same thing on your own data.

A far more interesting result comes from the same study. Before data models were added, both approaches performed moderately: 64,5% for text-to-SQL and 72,7% for the semantic layer. Adding just three models lifted both sharply. That is the strongest argument in the whole discussion: the work that genuinely improves answer quality is data modelling, not tool selection.

A semantic layer has one more benefit rarely mentioned in an AI context: it forces definitional disputes to be settled. To build one, somebody has to decide what revenue is. That is often the hardest part of the project and simultaneously the part that pays off whether or not anyone ever asks the model a question.

In the Microsoft ecosystem you do not have to build a semantic layer from scratch – it is already there and it is called the semantic model. Measures defined in DAX are exactly what definitions of metrics are in other stacks: one binding way to compute revenue, margin or turnover.

The difference between a company where Copilot in Power BI works well and one where it answers nonsense usually comes down to three things. First: whether figures are computed by measures or ad hoc inside visuals. Second: whether tables, columns and measures have their descriptions filled in, and whether those descriptions are written in business language rather than database shorthand. Third: whether the model contains synonyms – the fact that sales says „turnover” and the board says „sales” has to be recorded somewhere.

The practical order of work therefore looks different from what most projects assume. First a review of the semantic model and completion of descriptions for the area you want to start with – usually sales or finance. Then a list of twenty questions people actually ask. Only then do you switch the assistant on. The reverse order ends in the conclusion „Copilot does not work”, when what does not work is the model description.

It is also worth narrowing the starting scope. An assistant over one well-described area answers credibly; the same assistant over every model in the organization has a dozen similarly named tables to choose from and starts guessing.

Conversational analytics needs the same rigour as any other AI component in production: a set of questions with expected answers, run regularly after every change.

The set is built from history, not from imagination. Take twenty to fifty questions that reached analysts by email or chat over the last quarter and establish the correct answer for each – the one the area owner will sign off on. That is simultaneously a quality test and a list of definitions to complete in the model.

Measure two things separately. First: whether the number is right. Second, and more important: whether an ambiguous question prompted a clarifying question instead of a guess. An assistant that responds to a revenue question with „do you mean invoiced value or fulfilled orders” is worth more than an assistant with a higher raw score that never signals doubt.

Repeat the test after every change to the data model and every change of engine version. It is the same mechanism we describe under AI agent evaluation – and in analytics it matters especially, because a change in the semantic model can alter answers to questions seemingly unrelated to it.