DashBench
OpenDash's open accuracy benchmark and regression gate. How it's built, how answers are graded, how to run it, and the v0 baseline.
Every accuracy technique in OpenDash (the semantic layer, verified queries, repair loops, the answer gate, the learning loop) is only real if it can be measured. DashBench measures it, and it is built to be published openly: a reproducible alternative to vendor-run BI benchmarks, whose questions, schema, transcripts and rubric aren't public.
Principles
- Grade results, not SQL text. A case passes when the answer's numbers match the reference result within tolerance, however the SQL was written.
- Every number must be cited. A number that doesn't come from a query run never passes.
- Ambiguity is a skill. Cases test sensible defaults, saying which reading was chosen, asking when it matters, and refusing when a question can't be answered from the data.
- Held-out phrasings. Every question has a wording written by an outside author and never shown to the analyst, so the benchmark measures generalization rather than memorization.
- Measure the semantic layer's lift. Runs with and without definitions show what the semantic layer is worth.
- Reproducible. Seeded generators, fixed dates and pinned engines. Anyone can regenerate the data and re-run it.
What's in v0
72 questions on dash-saas, a generated B2B SaaS company with the traps real data has: trials, mid-month plan changes, refunds and credits, several currencies, test accounts, churn versus pause, and seats.
| Group | Questions | What it tests |
|---|---|---|
| Metric path | 17 | Dimensions, filters and periods over approved metrics |
| SQL over tables | 16 | Questions no metric covers |
| Readings and refusals | 6 | Ambiguous words, missing data, forecasts, personal data |
| Glossary | 6 | Words only the company's own glossary settles |
| Search | 3 | Rules only a guide holds, asked without the guide's words |
| Dialect traps | 3 | ClickHouse merges, integer division on Postgres, FILTER on MySQL |
Grading
Expected values come from reference SQL run on the seeded data at grading time. An answer passes when its cited values match (counts exactly, amounts within 0.5%) and its answer gate passed.
Must-never failures fail the run regardless of score: a typed number, an external effect without approval, a cross-tenant leak, raw SQL presented without the Unmodelled label, or a held-out phrasing leaking into the analyst's context.
The v0 baseline
Recorded on 2026-10-10 and 2026-10-11, with claude-sonnet-5-5 and the demo company's definitions approved:
| Run | Right | Untraced |
|---|---|---|
| Postgres, 60 questions (×2) | 60/60 | 0 |
| ClickHouse, 60 questions | 60/60 | 0 |
| MySQL 8.4, 66 with the glossary | 66/66 | 0 |
| Postgres, 69 with the glossary, guides and vectors | 69/69 | 0 |
| Postgres, 66 held-out wordings | 65/66 | 0 |
The one held-out miss was a gap in the knowledge, not the wording: two columns could answer "paid seats", and no metric settled which. A governed metric was added, and both wordings now take the metric path. That question's held-out wording no longer counts as blind.
What it doesn't show yet: one dataset of three, one held-out wording per question, and no nightly run (CI has no model key). The dash-plg and dash-commerce datasets, about 200 questions, more held-out wordings and the nightly job come next. The full record, every run, is in docs/evals/DASHBENCH.md in the repository.
By model
The 21-question golden set, with definitions and the glossary, on each Anthropic model the registry lists. OpenAI's and Google's models aren't measured yet.
| Model | Right | Cost for the set |
|---|---|---|
| Claude Sonnet 5.5 | 21/21 | $0.30 |
| Claude Haiku 5.5 | 21/21 | $0.02 |
| Claude Opus 5.5 | 21/21 | $0.67 |
| Claude Fable 5.1 | 21/21 | $1.91 |
The golden set doesn't separate them on correctness; cost and speed do. A harder set is what will show where the stronger models earn their price.
Run it
pnpm demo:seed # the dataset
pnpm golden # 21 questions, about 90 model calls
pnpm evals # all 72, on a fresh databaseUseful flags for pnpm evals:
| Flag | Runs |
|---|---|
--subset pr | 10 stable, cheap questions |
--subset glossary, --subset search, --subset traps | One group |
--dialect clickhouse | Against ClickHouse (or mysql) |
--wording holdout | The held-out wordings (--wording n for the n-th other wording) |
--without metrics | Schema only, no definitions |
--without glossary,guides,vectors | Without one or more kinds of knowledge, to measure their lift |
--model id | Another model (the default is Claude Haiku) |
--only id,id | Specific questions |
"Today" is pinned to 2026-10-10, so "last month" means September 2026 on any date. Reports go to packages/evals/reports/.
If your pull request changes prompts, tools, retrieval, the compiler or the answer gate, run pnpm golden or pnpm evals and include the score change.
What will be published
The generators and seeds, the questions and phrasings, the reference SQL, the graders and the harness, and results per release. Code under Apache 2.0, data under CC BY 4.0. Third parties can submit runs with transcripts. Only a rotating holdout stays private, so published scores stay meaningful.