osmosMind
← Leaderboard

SWE-Bench Verified

Ranked by Resolved %·8 models·Updated June 2026

#ModelScore
1
Anthropic

Claude Opus 4.8

Anthropic

79.4
2
OpenAI

GPT-5.4

OpenAI

76.1
3
Anthropic

Claude Sonnet 4.6

Anthropic

74.8
4
Google

Gemini 3 Pro

Google DeepMind

71.5
5
DeepSeek

DeepSeek V4

DeepSeek

66.2
6
AlibabaCloud

Qwen3 Max

Alibaba

62.7
7
Zhipu

GLM-5

Zhipu AI

58.9
8
Meta

Llama 4.1 405B

Meta

55.8

Scores are illustrative sample data.

What it measures

SWE-Bench Verified asks a model to resolve a real software issue inside a real repository. Each task ships with a runnable environment and the project's own test suite, so a patch is accepted only when the tests pass — the same bar a human contributor faces.

Methodology

  • Curated tasks — Issues with unambiguous acceptance criteria are drawn from real GitHub PRs across many languages.

  • Frozen repositories — Dependencies and the repo snapshot pin into a one-command, isolated environment that reproduces every run.

  • Test-gated scoring — The project's own test suite decides pass or fail, and every attempt logs its exact failing tests.

← Back to Leaderboard