Expert article

Talking to your company data: why text-to-SQL gives wrong numbers and what a semantic layer fixes

This article is for management boards, CFOs and managers who have been offered a way to “talk to their data” – asking AI questions about sales or costs in plain language. We explain where wrong answers come from that are not obvious at first glance. We also show what fixes them and how to check quality before the figures reach a board meeting.

Author: Kacper Włodarczyk, CEO of ALGORCOMPPublished: August 23, 2026Reading time: 11 min readData and analyticsFor: Mid-sized company
Talking to your company data: why text-to-SQL gives wrong numbers and what a semantic layer fixes

What “talking to your data” is, and why the demo works better than the rollout

Databases answer questions written in SQL, the language programs use to ask a database for data. Text-to-SQL means turning a question asked in plain language, such as “how much did we sell in June?”, into such a query. An AI model does this, and the result comes back as a number, a table or a chart.

In a demo this looks impressive, because the sample database has a few clearly structured tables. A real data warehouse – a database that collects data from the ERP, CRM and accounting systems for reporting – looks different. It has hundreds of tables, abbreviations in column names and several generations of solutions that nobody has removed.

The Spider 2.0 benchmark presented at the ICLR 2025 conference shows the difference well. An agent based on the o1-preview model solved only 21.3% of the tasks based on real company databases. On the academic Spider 1.0 dataset, GPT-4-based methods had previously reached 91.2%. The difference comes from the data the model works on and the complexity of the tasks.

Then there is joining tables. A question about sales to premium-segment customers needs one join in a well-organised database, but several in a database that has grown over years. Each join is a place where rows of data can be lost or duplicated.

What a silent mistake looks like

In an analysis of 4,602 incorrect queries (Shen et al., arXiv 2501.09310), almost one error in three concerned the meaning of the question or the data rather than query syntax. This is an important distinction. A syntax error ends with a message that cannot be missed. An error of meaning ends with a number that looks correct.

Below are three typical kinds of such mistakes. They all arise because rules that are obvious to people in the company are not written down anywhere in the database, so the model does not know them.

  • Several definitions of the same term. The question “what was revenue in June?” has at least three correct answers in the database. These are the value of invoices issued in June, the value of orders fulfilled in June and revenue recognised in the accounts. The model picks the first field that fits and does not say which it chose.
  • Filters nobody mentions. Everyone in the company knows that cancelled documents are not counted, corrections go into the month of the original invoice and the subsidiary has a separate report. These rules live in a few people’s heads and in the code of old reports, but not in the database.
  • Units and currencies. A price column with no information on whether the amount is net or gross, in which currency and at what exchange rate, is just a number to the model. The total looks credible but is wrong.
Talking to your company data: why text-to-SQL gives wrong numbers and what a semantic layer fixes

What a semantic layer fixes

A semantic layer is the company’s written dictionary, sitting between the database and the person asking. It contains a list of metrics, each with a single way of calculating it, and the dimensions by which they may be broken down, such as region or product group. It also holds descriptions in the language the company uses. The model then does not write calculations from scratch but chooses from ready-made, approved terms.

In the dbt Labs test from April 2026, however, the biggest gain came from tidying up the data itself. Adding three data models raised text-to-SQL from 64.5% to 90.0% for Claude Sonnet 4.6. The semantic layer added further points on top, as the table below shows.

You need to know the scale of the test: 11 questions, each asked 20 times, on a single insurance dataset with 15 tables. It is a strong signal of direction, not proof that you will get the same result. The same test also showed a limitation: a semantic layer answers only the questions its definitions cover.

Building a semantic layer has one more advantage. It forces you to settle disputes over definitions, such as what counts as revenue. This is often the hardest part of the project, but it is valuable even without AI.

Share of correct answers in the dbt Labs test, April 2026 (11 questions × 20 repetitions)
VariantText-to-SQLSemantic layer
Raw tables (both models)64.5%72.7%
After adding 3 data models – Claude Sonnet 4.690.0%98.2%
After adding 3 data models – GPT-5.3-Codex84.1%100%

What this looks like in Power BI

In Power BI, the semantic layer already exists and is called a semantic model. Measures – calculation formulas stored in the model – are exactly that: a single agreed way of calculating revenue, margin or stock turnover.

Whether Copilot in Power BI gives sensible answers usually depends on the few things listed below. Microsoft provides a tool for this called “Prep data for AI”, in which the model owner records instructions for the AI and approved answers to common questions.

Copilot in Power BI requires Fabric F2 or Premium P1 capacity or higher; a Pro licence alone is not enough. According to the documentation, some features do not yet officially support questions in languages other than English. Test this with questions in Polish before making it available to the management board.

  • Whether figures are calculated with measures in the model or ad hoc in individual charts.
  • Whether tables, columns and measures have descriptions written in the company’s language rather than database abbreviations.
  • Whether the model contains synonyms – salespeople say “turnover”, the management board says “sales”, and they mean the same thing.
  • Whether the assistant has a narrow scope. Over one well-described model it answers reliably; over all the models in the company it has many similar tables to choose from and starts guessing.
  • Whether approved answers have been recorded for the most common questions.
A question asked in plain language being turned into a database query using a single agreed definition of a metric

An error in a query ends with an error message. An error in meaning ends with a number that looks exactly as it should.

How to test the assistant before it reaches the management board

Build the test set from history, not from imagination. Take a few dozen questions that were sent to analysts by email or Teams in the last quarter. For each one, establish the correct answer and have it signed off by the person responsible for that area.

Measure two things separately. First: is the number right? Second, and more important: when the question was ambiguous, did the system ask for clarification rather than guess?

An assistant that responds to a question about revenue with “do you mean invoiced value or fulfilled orders?” is worth more than one with a higher score that never flags any doubt. Asking for clarification is a strength, not a weakness.

Repeat the test after every change to the data model and every change of AI model version. Changing a single measure can change the answers to questions that seem unrelated to it.

Where to start: a plan for one area

The most common mistake is the order of steps: first switching on the assistant, then trying to understand why its answers make no sense. The conclusion is then “AI doesn’t work”, when in fact it is the description of the data that does not work.

The order below reduces this risk. The second step usually takes the most time, because it requires several departments to agree rather than technical work.

  • Choose the one area with the most questions – usually sales or finance.
  • Collect a list of real questions and agree the definitions of the terms used in them.
  • Review the data model: replace calculations in charts with measures, and add descriptions and synonyms.
  • Prepare a test set with approved answers and test the assistant against it.
  • Make the assistant available first to people who can judge the results, such as analysts and financial controllers.
  • Only after a few weeks of fixes, make it available more widely, with clear information on which questions it handles.

When text-to-SQL is enough, and when it is better not to use it

Text-to-SQL without a semantic layer has sensible uses. It works for ad hoc analysis on smaller, well-understood datasets, when an analyst is at the computer to judge the result and correct the query.

The problem starts when the answer goes straight to someone who has no way of checking where the number came from. This applies especially to the management board, the supervisory board and customers. They need approved definitions, not queries written from scratch each time.

Data access also calls for caution. The query runs with the permissions of the account the tool uses. If that account can see the whole database, anyone asking a question may receive, for example, salary data.

Related topics in the knowledge base

Read next

FAQ

Talking to your data – frequently asked questions

Questions that come up when a company is considering an AI assistant for data analysis.

Will a newer AI model solve the problem without any work on the data?
No. In the tests cited, the biggest improvement came from tidying up the data, not from changing the model. A newer model may handle difficult questions better, but it will not guess which of three definitions of revenue your management board has adopted.
Will the assistant replace reports?
It is more likely to complement them. Reports answer recurring, planned questions; the assistant answers one-off, unforeseen ones. A well-prepared assistant reduces the number of requests for one-off summaries and gives analysts time for work that requires interpretation.
How much work does it take to prepare one area?
That depends mainly on how many disputes there are over definitions and what state the data model is in. If the company has a working Power BI model with measures, what is usually missing are descriptions and synonyms. If figures are produced in spreadsheets, a model has to be built first.
Does company data go to the AI model provider?
The model receives the question and the part of the data needed to answer it. Check in the tool’s documentation in which region the data is processed and whether the provider may use it for training. In Power BI, the administrator has a setting that determines whether data sent to the AI model may be processed outside the company’s region.
How can you tell that the assistant is guessing?
Good tools show which measure or table a number came from. In Power BI, there is a feature that explains how Copilot arrived at its answer. If a tool does not let you check where a number came from, do not make it available to people who make decisions based on it.

About this page

About the author

Kacper Włodarczyk — CEO of ALGORCOMP

Kacper Włodarczyk

CEO of ALGORCOMP

Kacper specializes in implementing Microsoft 365 Copilot, Copilot Studio, Power Platform (Power Automate, Power Apps, SharePoint) and AI agents for mid-sized B2B companies in Poland. He leads dozens of projects covering AI strategy, Power Platform governance, document workflow automation and sales processes. In his writing he focuses on the practical side of AI adoption in organizations — from the first proof of concept to company-wide rollout — with particular attention to data security, compliance (GDPR, NIS2, the AI Act) and return on investment.

Meet the team

Want to check whether your data is ready for questions put to AI?

In a free consultation we will review the data model for one area, identify missing definitions and descriptions and suggest a set of test questions.

Featured

Related articles