Evaluation · Frozen vs research
Frozen baseline (v1) vs V2 (100 tasks). Source: benchmarks/baseline/summary.json (seed 42, 20 datasets) and benchmarks/v2/catalog.json (30 datasets, 11 categories).
Tasks
50
Task success
100%
Evidence
100%
Mean latency
142.9ms
SQL 1 · Unsupported 0
100 tasks · 30 datasets · 11 categories
Tasks
100
Datasets
30
EDA 11 · SQL 11 · Statistics 13 · Regression 9 · Classification 9 · Time Series 8 · Visualization 8 · Data Quality 8 · Data Profiling 6 · Clustering 7 · Evidence Validation 10
Task success · evidence coverage · SQL accuracy (v1 frozen)
100 tasks across 11 categories
8 categories · task success rate (frozen baseline)
uv run dsa --catalog benchmarks/v2/catalog.json --datasets benchmarks/v2/datasets --limit 100 --out /tmp/v2-bench