Coding benchmarks: Which model writes the best code?

Coding benchmarks: Which model writes the best code?

Analysis

If you manage a team that ships software, you have probably watched the parade of AI coding models and wondered which one is worth paying for. The marketing decks all say the same thing. Every model is the best at coding. They cannot all be right.

That is the problem SWE-bench Pro was built to solve. Instead of asking a model to autocomplete a function, it hands the model real tickets from real codebases, 1,865 tasks pulled from 41 repositories across Python, Go, TypeScript and JavaScript, and checks whether the code actually works (Scale Labs (opens in a new tab)). It is closer to "can this thing do a junior engineer's day" than "can it pass a quiz."

But there is a catch worth knowing before you read a single percentage. Most of the headline scores come from the model makers themselves, run on their own setups. Neutral, apples-to-apples scores tend to land a fair bit lower. So treat the leaderboard below as a ranking with an asterisk, and we will point out where the vendor number and the standardised number part ways.

The short version for a busy team: Claude Opus 4.8 is the strongest model you can actually use right now, the open-weights field has gotten genuinely good and cheap, and the difference between the top closed model and a budget open one is smaller than the price tags suggest.

The SWE-bench Pro leaderboard

A word on that table before you act on it. The numbers above are mostly vendor-reported, meaning each company ran the test on its own tooling. Scale's standardised harness, which runs every model the same way, tells a less flattering story: its neutral leader is GPT-5.4 (xHigh) at 59.1%, well short of the vendor figures you see here (Scale Labs (opens in a new tab)). Same benchmark, different plumbing, very different result. Read the ranking as "roughly who's ahead," not as a precise score you can quote to your CFO.

A few rows also deserve their own asterisks. Fable 5's chart-topping 80.3% comes from Anthropic's own launch materials using Anthropic's own scaffolding, and it has been called contested by independent reviewers; standardised leaderboards paint a more competitive picture (Morph LLM (opens in a new tab)). The table's 54.2% for Gemini 3.1 Pro is also higher than the 46.1% standardised figure that turns up in the source data. And several rows, DeepSeek V3.5, GLM-5.2, Llama 4, Grok 4, Kimi K2.7-Code, Mistral Large 2, Qwen 3, GPT-5.5 Pro, GPT-5.5 Instant and Opus 4.7, could not be confirmed against the leaderboards we checked, so treat their exact percentages and prices as unconfirmed. One likely version mix-up worth flagging: the sources reference GLM-5.1 at 58.4%, not GLM-5.2.

Opus 4.8's figures, by contrast, hold up: 69.2% vendor-reported, $5.00 input / $25.00 output per million tokens, 1M context, released 28 May 2026 (Finout (opens in a new tab)). MiniMax M3's 59.0% also checks out, along with its 1M context and roughly $0.30/$1.20 launch pricing (Fello AI (opens in a new tab)). GPT-5.5's 58.6% is corroborated across several sources too (Morph LLM (opens in a new tab)).

Tier analysis

The tiers below are our reading of the numbers, not an official published ranking. They are a sensible way to group models by what they can realistically handle, but they sit on top of scores that carry the caveats above.

Tier 1 (65%+): Claude Fable 5 and Opus 4.8. On these numbers, they are the only two that reliably get through complex, multi-file engineering work. And there is a twist: a US export-control directive on 12 June 2026 forced Anthropic to suspend Fable 5 and Mythos 5 for everyone. The order required cutting off foreign nationals, and since nationality cannot be checked in real time, both models went dark for all users while Opus 4.8, Sonnet 4.6 and Haiku 4.5 kept running (BetaNews (opens in a new tab)). That leaves Opus 4.8 as the only Tier 1 model you can actually log in and use.

Tier 2 (55-65%): Opus 4.7, GPT-5.5 Pro, MiniMax M3, Sonnet 4.6, GPT-5.5, Kimi K2.7-Code. These handle most coding work fine but start to slip on the gnarliest edge cases. MiniMax M3 and Kimi K2.7-Code are the open-weights standouts here, and M3 in particular punches above its price.

Tier 3 (45-55%): Grok 4, Gemini 3.1 Pro, DeepSeek V3.5, GLM-5.2, Llama 4, Gemini 3.5 Flash, Mistral Large 2. Fine for routine work, boilerplate, simple debugging, documentation, but not something you'd trust with a hard problem unsupervised.

Tier 4 (<45%): Qwen 3, GPT-5.5 Instant. Basic help only. Good for explaining code or knocking out a small script, not for production engineering.

Price-per-point analysis

If you care about getting the most capability per dollar, here is how the value picks shake out. Same caveat applies, these are our derivations from the scores, not a published value index.

1. Llama 4, Free, 50.2% (effectively unlimited value if you've got the GPUs to run it)
2. DeepSeek V3.5, $0.15/$0.60, 52.4% (the best value among paid models)
3. MiniMax M3, $0.30/$1.20, 59.0% (the best open-weights coder, with a note: open weights were committed at launch but reportedly hadn't shipped as of reporting)
4. Gemini 3.5 Flash, $0.35/$0.70, 48.2% (the best of the budget closed models)

Verdict

If you want the most coding capability you can actually access today, Claude Opus 4.8 (69.2%) is the pick, Fable 5 sits higher on paper but is offline. For the best open-weights option, MiniMax M3 (59.0%) leads. For value, DeepSeek V3.5 (52.4%) or Llama 4 (50.2%, free) get you most of the way at a fraction of the cost. And for serious engineering work, skip GPT-5.5 Instant and Qwen 3.

One last reminder: every number here carries the vendor-versus-standardised gap. Before you commit a team to a model, run it against your own codebase on the kind of tickets you actually close. The leaderboard tells you who to shortlist. Your repo tells you who to hire.

Coding benchmarks: answer-first summary

Coding benchmarks matters because it can change how Founders and operators plan, build, or govern an tool evaluation workflow. We rank 17 models by SWE-bench Pro.

The direct answer is this: do not treat the topic as a standalone trend. Treat it as a decision about inputs, outputs, review ownership, data exposure, and whether the workflow produces a result that is faster, safer, or more useful than the current process.

Coding benchmarks: implementation checklist

• Define the user, job to be done, and success metric for the tool evaluation workflow.
• Collect real examples, policies, source files, customer questions, or search queries before writing prompts or choosing tools.
• Separate low-risk drafts from decisions that need approval, privacy checks, or senior review.
• Document what the AI is allowed to access, what it must not access, and who signs off before production use.
• Review time to value, adoption rate, cost per workflow, quality review score after a small pilot rather than judging the idea from a demo.

This keeps the work practical. It also gives search engines and AI answer engines a clean factual structure: what the topic is, who it helps, what to do next, and which risks matter before implementation.

Decision criteria for Coding benchmarks

Practical example for Coding benchmarks

A small business could use this article to choose one practical test. For example, a manager might take one customer-facing process, one internal document workflow, or one recurring content task and redesign only that step with AI support. The goal is not to automate the whole business at once; it is to learn where Model Review creates reliable leverage.

The useful deliverable is a short operating note: the trigger, the source material, the prompt or tool, the review checklist, the escalation rule, and the metric. That note becomes the handover asset for staff training, SEO/GEO content, service delivery, or future agent work.

Risks and controls for Coding benchmarks

The common failure pattern is moving too quickly from a promising idea into an unmanaged workflow. For Coding benchmarks, the risk is not only bad output. It can also be unclear data permission, staff confusion, duplicate content, unreviewed customer advice, or a tool that quietly changes cost or capability.

• Control tool sprawl with a named owner, a review step, and written acceptance criteria.
• Control unclear pricing with a named owner, a review step, and written acceptance criteria.
• Control vendor lock-in with a named owner, a review step, and written acceptance criteria.
• Control unreviewed data sharing with a named owner, a review step, and written acceptance criteria.

Measurement plan for Coding benchmarks

A useful AI or SEO initiative should leave evidence. Track time to value, adoption rate, cost per workflow, quality review score and compare the pilot against the current process. If the measure does not improve, keep the learning but avoid scaling the workflow.

For GEO readiness, the page should also answer the core question directly, define the entities involved, include implementation steps, explain tradeoffs, and link readers to the next relevant AI Kick Start service, guide, tool, or article.

Definitions and entities for Coding benchmarks

For search, GEO, and staff handover, define the core entities in plain language. In this article the important entities are the workflow owner, the AI tool or model, the source material, the review process, the risk boundary, and the measurable business outcome. Clear definitions make the page easier for people to scan and easier for AI answer engines to quote accurately.

• Workflow owner: the person accountable for deciding whether Coding benchmarks belongs in the business process.
• Source material: the documents, examples, policies, URLs, prompts, videos, or customer questions that ground the output.
• Review boundary: the point where a human checks accuracy, privacy, brand voice, or customer impact before the result is used.
• Success metric: the measure that proves whether the tool evaluation workflow is worth repeating.

Coding benchmarks versus doing nothing

Doing nothing is also a decision. The cost may be slow manual work, weaker search visibility, inconsistent advice, duplicated effort, or staff using unmanaged AI tools without a shared process. The practical question is whether a controlled pilot can reduce that cost without creating a larger governance problem.

AI Kick Start handover package for Coding benchmarks

A production handover should be concrete enough that another person can run it. For Coding benchmarks, that means a short brief, a workflow map, approved prompts or tool settings, source material, a review checklist, internal links to supporting resources, and a simple measurement sheet. This is the difference between reading about AI and turning it into operational capability.

That packaging also strengthens E-E-A-T. It shows experience through implementation notes, expertise through decision criteria, authoritativeness through source-aware structure, and trust through risks, controls, and review steps. The article becomes useful even if the reader never buys a tool because it helps them make a better operational decision.

Related AI Kick Start paths for Coding benchmarks

Readers who need help moving from article to implementation should connect this topic to related AI Kick Start services such as AI automation, agent systems, secure local AI, SEO/GEO growth, practical training, or a workflow audit. That keeps the page useful for humans and gives crawlers a clearer topical map across the site. It also gives AI answer engines a cleaner relationship between the article, the service category, the local Australian business context, and the next practical step a reader can take.

Source notes: AI automation for small business, Agent systems, SEO and GEO growth, AI resources