osmosMind

Data/Software Eng.

SWE Engineering Tasks

Software-engineering tasks drawn from real codebases. Each instance ships a runnable repo environment, the gold patch, the test patch, and an evaluation script — so a candidate fix is judged pass or fail against real FAIL_TO_PASS / PASS_TO_PASS tests, not toy problems.

Family

  • SWE-Bench Multilingual

    Issue fixes across 8+ language ecosystems — Rust, Java, Go, PHP, Ruby, JS/TS, C — well beyond Python-only sets.

  • SWE-Bench Verified

    Human-validated, self-contained instances whose problem statements carry enough context to be solvable.

  • SWE-Bench Pro

    Enterprise-grade difficulty: larger repositories, longer multi-file patches, and stricter acceptance criteria.

  • SWE-Bench Multimodal

    Tasks that pair the codebase with visual or UI context, so a fix must reconcile rendered behaviour with source.

  • SWE-Bench Live

    Freshly curated, contamination-free instances refreshed on a rolling basis to resist training-set leakage.

Coverage

  • GitHub PRs
  • Issue fixes
  • 8+ language ecosystems
  • Test repair
  • Real engineering constraints

Deliverables

  • Task package (JSONL)
  • Per-instance repo environment
  • Gold & test patches
  • Evaluation scripts
  • Pass-rate records & acceptance report

The pipeline

From source to acceptance

We don't hand-label a pile and ship it. Every category moves through a closed, instrumented loop — generated to a brief, checked by machines, adjudicated by experts, and traceable end to end — but the path each data type takes is its own.

  1. 01

    Task Sourcing

    Tasks are drawn from real merged pull requests and their linked issues, inheriting genuine engineering constraints and acceptance criteria that were settled by working maintainers.

  2. 02

    Environment Freeze

    Each instance's dependencies and repository snapshot are frozen into a single-command reproducible environment, so a candidate fix is always evaluated against the exact same ground truth.

  3. 03

    Oracle Construction

    The gold patch and its accompanying test patch are reconstructed to form a runnable oracle, defining precisely which previously failing tests a correct solution must turn green.

  4. 04

    Automatic Verification

    An evaluation harness applies a candidate patch and runs the fail-to-pass and regression suites on hidden tests, settling pass or fail objectively rather than by human judgement.

  5. 05

    Coverage Check

    The set is audited for balance across languages and ecosystems, so measured ability reflects general engineering skill rather than fluency in a single stack.

  6. 06

    Acceptance Report

    A final stratified sample and acceptance report confirm that every instance builds, verifies, and traces back to its originating change before the package ships.

Every run emits a learning signal that feeds back into the source set — the pipeline tightens itself, batch over batch.

A specimen

See the data itself

One real, trimmed sample from this category — the scenarios it serves, why it matters for training, and the shape of the data as delivered.

Where it’s used

  • Evaluating patch-generation and issue-resolution ability
  • Training on real multi-file bug fixes with test oracles
  • Regression-safe RL from verifiable pass/fail rewards

Why it matters for training

Critical

The gold standard for measurable coding ability: every instance has a real test oracle, so reward is objective rather than judged.

Notable features

  • 50 hard instances
  • 8+ language ecosystems
  • FAIL_TO_PASS test oracles
  • Per-instance Docker env
SWE_hardDelivery package
  • catalog.json21.8 KB
  • instances/
  • pallets-flask-5917-3e16c3fc/
  • evaluation.sh1.2 KB
  • environment/
  • Dockerfile0.9 KB
  • materials/
  • evidence/
  • fastify-fastify-6716-01c9f40a/
  • tokio-rs-tokio-pr8131-me-4faf89d2/
  • symfony-symfony-64261-55a9b0ea/
  • grpc-grpc-go-9102-15bfd03d/
  • rubocop-rubocop-15139-19b52bd4/
README.mdmarkdown
# SWE Hard 50

50 high-difficulty software-engineering instances spanning Python,
JavaScript, Java, Rust, Go, C#, PHP, and Ruby. For patch-generation,
code-repair, and issue-resolution evaluation.

Each instance provides a task description, gold code patch, test patch,
environment build materials, and an evaluation script — for independent
inspection, reproduction, and re-integration.

- dataset.jsonl — all 50 instances, one JSON per line (primary index)
- catalog.json — dataset catalogue and statistics
- instances/<id>/task.json — full structured record
- instances/<id>/evaluation.sh — evaluation script
- instances/<id>/environment/Dockerfile — environment build

Need a sample or a custom build?

Tell us your spec and scale — we deliver to order.

Get in Touch
← Back to Data