If you’re trying to figure out which LLM is best at writing software, you’ll eventually land on ArtificialAnalysis.ai. It looks authoritative. Clean design, lots of models, comparison charts — exactly what you want when you’re evaluating options. They publish a “Coding Index” that ranks models on programming ability.
There’s just one problem: Artificial Analysis’s Coding Index doesn’t measure coding ability. Not even close. And by publishing it under that name, they are actively misleading every developer, team lead and decision-maker who relies on it.
What’s Actually Being Measured
The Coding Index is a composite of two benchmarks: Terminal-Bench Hard and SciCode. Let’s look at what each one actually tests.
Terminal-Bench Hard evaluates AI capabilities in terminal environments — system administration, data processing and software engineering tasks. In practice this is primarily sysadmin work. Can the model navigate a filesystem, run commands, compile things? These are useful skills in the same way that knowing how to use a screwdriver is useful if you’re a building architect. It’s table stakes, not a measure of the thing that matters.
SciCode is a scientist-curated benchmark with 288 subproblems across 16 scientific disciplines. The benchmark’s own description says it “requires integrating scientific knowledge with programming skills to solve real research problems.” Read that again — it’s explicitly testing the intersection of domain-specific scientific knowledge and coding. If a model happens to know less about computational fluid dynamics but writes better production software, it scores lower on the “Coding Index.” A model that’s brilliant at architecting maintainable systems but doesn’t know the Navier-Stokes equations gets penalized on what’s supposed to be a coding benchmark.
That’s it. Two benchmarks. Sysadmin tasks and science homework. That’s what Artificial Analysis chose as the entire basis for ranking models on coding ability.
What’s Missing
Here’s what the Coding Index doesn’t include — benchmarks that actually measure software development:
SWE-bench Verified, where models patch real bugs in real open-source repositories with real test suites. It’s the closest thing the industry has to measuring actual software engineering: read an existing codebase, understand the problem in context, produce a working fix.
Aider’s polyglot benchmarks, LiveCodeBench, BigCodeBench and EvalPlus — all of which test code generation directly against functional correctness.
Beyond specific benchmarks, the Coding Index doesn’t test for any agentic capabilities, complex problem solving or long-horizon task completion — all of which are directly relevant to how models are actually used for software engineering today. The ability to plan an approach, execute across multiple files, recover from errors and sustain coherent work over extended sessions is increasingly what separates useful coding models from toys. The Coding Index measures none of it.
None of these are perfect. But they at least involve writing code that has to work.
Why This Matters
When I looked at Artificial Analysis’s Coding Index chart, it ranked Claude Sonnet 4.6 above Claude Opus 4.6 for coding. I use both models extensively every day for real software development work. That ranking is simply wrong. Sonnet is a capable model, but Opus is substantially better on complex software — multi-file refactors, subtle architectural issues, large-context reasoning across interconnected systems. The kind of work that actually defines whether a model is useful for serious development.
Artificial Analysis’s benchmark can’t see this because their benchmark isn’t measuring it.
The real danger isn’t that experienced developers will be misled — most of us will smell the problem quickly, just as I did. The danger is that decision-makers, team leads evaluating tools and developers earlier in their careers will see “Coding Index” on a professional-looking site and reasonably assume it reflects which model will best help them build software. Artificial Analysis is failing those people. The Coding Index is a grab bag of tangentially related capabilities wearing a label that implies something it absolutely does not deliver.
The Takeaway
If you’re evaluating models for software development, don’t use Artificial Analysis’s Coding Index. Look at SWE-bench Verified and Aider’s benchmarks. Look at what people doing real work with these models are reporting. Better yet — try them yourself on your actual tasks.
Composite indexes with authoritative-sounding names are convenient. But convenience is worthless when the underlying data doesn’t measure what the label claims. Artificial Analysis should either fix their Coding Index to include benchmarks that actually test software development or stop calling it a Coding Index. What they’re publishing right now is misinformation with a clean UI.
Be First to Comment