June 7, 2026
A model can top SWE-bench and still stumble on your codebase within an hour, inventing conventions you never asked for and guessing where it should have asked a question.
Why it matters: Teams keep picking coding tools off leaderboard rank, then discover the score measured a different job than the one they actually need done.
The Details:
Benchmarks hand a model a clean repo, a well-specified issue, and a passing test suite as the finish line. Real work looks nothing like that: a half-described bug, three competing patterns for the same task, a spec scattered across Slack messages and someone's memory. Passing a closed problem with a verifier attached is a real skill, but it may cover only a fraction of what a working developer does daily.
Labs chase leaderboard position because it is the one thing everyone can point to in a forum thread or a procurement meeting. That legibility pressure pulls development effort toward tasks that are easy to score and away from the messier judgment calls, like knowing when to stop and check with a human, that never show up on a public chart.
The fix is not building your own formal eval, which eats weeks and still misses unanticipated failure modes. Instead, run the tool on real tickets for two or three weeks and keep a lightweight log: what you asked, what it did, where you stepped in. Patterns like strong first-pass output but poor debugging recovery emerge fast and are far more actionable than a score.
Two traps sink this process. Inertia kills the log unless friction is near zero, so a shared chat channel beats a spreadsheet. Sunk cost quietly reframes failures as "needed better context" once a team has already committed budget to a tool, so someone should be assigned to challenge that framing directly.
Bottom Line: A leaderboard can shortlist candidates, but only watching a tool work on your own code for a few weeks tells you if it will actually help.
Enjoy this article?
Listen to the Claude Code Conversations radio show or join the community.
