Off-the-Shelf Benchmark Packages
Plug-and-play evaluation assets aligned to the major public benchmarks. Where the categories above are built to spec, this is the procurement lane: standardized packages that match each benchmark's official task definitions and scoring, shipped with a reproducible validation environment so your team skips the reproduction work.
Coverage
- Terminal-Bench
- SWE-Bench
- health_bench
- STEM tracks
- Immediate-use packaging
Deliverables
- Data package
- Version notes
- Scoring scripts
- Validation environment
- Acceptance report
From source to acceptance
We don't hand-label a pile and ship it. Every category moves through a closed, instrumented loop — generated to a brief, checked by machines, adjudicated by experts, and traceable end to end — but the path each data type takes is its own.
- 01
Definition Parsing
Each benchmark's official task definitions and scoring rules are parsed in full, so the package is built against the source specification rather than a secondhand approximation.
- 02
Schema Mirroring
The tasks are mirrored into a matching, well-documented schema that preserves the official structure while making the assets clean to load and integrate into an existing harness.
- 03
Validation Environment
A runnable validation environment is assembled so the scoring pipeline executes end to end, letting a buyer reproduce results in place rather than reconstructing the harness themselves.
- 04
Cross-Environment Fidelity
Scores are reproduced in a second, independent environment and checked against the published official numbers, confirming the package scores exactly as the benchmark intends.
- 05
Versioning & Acceptance
Version notes and an acceptance report record the benchmark revision, any deviations, and the fidelity checks passed, so every asset traces cleanly back to its official source.
Every run emits a learning signal that feeds back into the source set — the pipeline tightens itself, batch over batch.
See the data itself
One real, trimmed sample from this category — the scenarios it serves, why it matters for training, and the shape of the data as delivered.
Where it’s used
- Standing up a standard eval harness quickly
- Procuring validated public-benchmark assets off the shelf
- Baselining against official benchmark numbers
Why it matters for training
Foundational
A fast, low-risk on-ramp: standardized assets you can buy today rather than reproduce yourself.
Notable features
This lane packages the public benchmarks that the SWE and Terminal categories above are built against — delivered as standardized, versioned assets with scoring scripts and a validation environment. Explore those two categories to see the underlying data shape; here the difference is procurement, not format.
Need a sample or a custom build?
Tell us your spec and scale — we deliver to order.
← Back to Data