hojas/interesting/llms-are-bad-at-real-world-sql.md

LLMs are bad at real world SQL

https://cacm.acm.org/blogcacm/if-you-think-you-can-do-real-world-text-to-sql/

An interesting article from an academic blog that explains something I had sensed for quite a long time: LLMs may seem to perform a decent job at translating natural language requests into SQL queries. But that only holds on pristine, lab-grade database schemas. The moment you jump into a messy, real database, it all breaks down.

This guys replicated some benchmarks on a nasty DWH database and suddenly, success rates went down from 80% of cases to an astounding... 0%. Way to go Claude.