77.1% on BIRD dev with one model call per question, and a test-set submission in
BIRD is the text-to-SQL benchmark the serious systems compete on: 1,534 dev questions over 11 real databases, scored by execution accuracy, with a hidden test set that only the BIRD team can run. I ran schemagate on the full dev set this week and sent the test-set submission. Dev set, all 1,534 questions, BIRD's own evaluator: 77.12% execution accuracy (simple 80.97, moderate 71.34, challenging 71.03), with Claude Opus 5.5. What makes that number interesting is the cost. Per question: schemagate reflects the database and picks the tables the question needs; the model sees only those tables' DDL and BIRD's column notes; the model writes one query; if SQLite errors, it sees the error and fixes it (that happened once in 1,534). That's one model call per question, about 2,100 prompt tokens, no candidate generation, no voting, no agent loop. The systems at the top of the main leaderboard (82.95 on the test set) typically spend 5–20 calls per question. We're submitting to find out wher
BIRD is the text-to-SQL benchmark the serious systems compete on: 1,534 dev questions over 11 real databases, scored by execution accuracy, with a hidden test set that only the BIRD team can run. I ran schemagate on the full dev set this week and sent the test-set submission.
Dev set, all 1,534 questions, BIRD's own evaluator: 77.12% execution accuracy (simple 80.97, moderate 71.34, challenging 71.03), with Claude Opus 5.5.
What makes that number interesting is the cost. Per question:
- schemagate reflects the database and picks the tables the question needs;
- the model sees only those tables' DDL and BIRD's column notes;
- the model writes one query; if SQLite errors, it sees the error and fixes it (that happened once in 1,534).
That's one model call per question, about 2,100 prompt tokens, no candidate generation, no voting, no agent loop. The systems at the top of the main leaderboard (82.95 on the test set) typically spend 5–20 calls per question. We're submitting to find out where one call lands on the hidden test set; BIRD's turnaround for API-based methods is 1–5 days, and I'll post the verified number either way.
Three things worth knowing before you compare:
- A re-run of the same code gave 80.20 on Mini-Dev where the first run gave 81.00. Opus is not deterministic run to run; treat single-run numbers as ±1.
- BIRD's dev set has known annotation errors (Jin et al., VLDB 2026 found problems in about half of Mini-Dev's reference answers). Every entry is scored against the same answers, so rankings hold, but absolute numbers are softer than they look.
- The leaderboard is self-reported for dev and Mini-Dev. Our Mini-Dev rows (81.0 / 72.6 / 72.4 on SQLite / PostgreSQL / MySQL, above the current #1 in each) are in PR #251 awaiting the maintainers. The test set is the only part BIRD runs itself.
Why bother with a one-call system at all? Because schemagate's job isn't the leaderboard; it's making sure a caller never sees a table they can't read, and never gets a query run that touches one. On the access-control benchmarks (RBAC Text-to-SQL, Spider-ACL) it has the best published results, with zero ungranted reads in 8,128 queries. The BIRD number shows that layer costs nothing in accuracy.
Predictions, evaluator output and the exact submission package: https://github.com/ashishsinha1602/schemagate/tree/main/benchmarks/results/bird_dev
