Pitfalls in natural-language-to-SQL

Natural-language-to-SQL looks easy in a demo. You ask a question, the model writes a query, and a table appears. The hard part is that a wrong query often looks right.

The query runs but answers the wrong question

The most dangerous failure is a valid query that returns plausible numbers. The model may join on the wrong key, double count rows after a join, or filter on the wrong date column. Nothing errors, so nobody notices.

Business terms are not in the schema

"Active customer", "revenue" and "last quarter" all have company-specific meanings. The schema does not say which column or rule defines them. If the model has to guess, it will guess differently on different days.

Schemas are too big

A real database can have hundreds of tables with unclear names. Putting all of it into the prompt is expensive and confuses the model. It helps to give the model only the relevant tables, found by search or by a curated layer.

Ambiguous questions

"Show me sales by region" depends on what counts as a region and which period is meant. Sometimes the right move is to ask a clarifying question instead of answering.

What helps

• A curated semantic layer that defines metrics once
• A short list of approved tables and columns, with descriptions
• Worked examples of questions and correct queries
• Showing the generated SQL next to the answer so a person can check it
• Validation before execution, covered in the next post

None of these removes the risk. They make errors rarer and easier to see.