Parsimony ranks coding models by how much code they add, without running any of it
A benchmark that re-reads the patches other benchmarks already published and counts syntax units added and removed. Its own data shows why that is tricky.
Coding benchmarks tell you whether a model fixed the bug and what it cost in tokens. They do not tell you whether it fixed the bug in six lines or sixty. Parsimony, by La5u, tries to measure that, and it does so without generating a single new model output.
The method is the story here. So is the finding that cuts against the premise, which the author publishes anyway.
How it works
Benchmarks such as SWE-bench Verified and DeepSWE publish the patches each model submitted. Parsimony takes those patches, applies each one to the task’s code at the right commit, and parses every changed file before and after. Nothing is executed. Pass or fail is taken from the benchmark’s own published results.
Because the inputs are public and nothing is run, the README says anyone can recompute every number.
It does not count lines, since a model could score well by cramming. It counts what the author calls coding units: statements, calls, comparisons, names, and literals. Comments, formatting, punctuation, and the length of a name count for nothing. The method document gives if x.count(y) > 0: foo(a, b) as 12 units.
The before and after unit sequences are diffed, and the score is units added minus units removed. The README’s example: replacing a four-line summing loop with return sum(items) removes 13 units and adds 3, for a net of minus 10.
Models are ranked by mean net units on the tasks they solved, smallest first. Tests, docs, and generated files are excluded. There are separate boards for Python, JavaScript, TypeScript, Go, and PHP, because unit counts do not compare across languages. The largest board covers 35 models on SWE-bench Verified’s 500 tasks.
Where it gets awkward
The project site reports that on boards with 26 or more models, the amount of code written and the number of tasks solved are positively correlated, with a Spearman coefficient between 0.33 and 0.60. Models that write more code tend to solve more tasks.
So a leaderboard sorted by smallness is partly sorted by something else. The author says so: smaller is not always better, and units measure size, not readability or design.
The method document lists further weaknesses. Rewarding net units rewards deletion. A hardcoded branch aimed at the test can cost fewer units than a general fix. And the agent wrapped around the model changes how much code comes out, even with the model held constant.
There is also a gap at the top. The newest models have no public patches, so the author ran some of them on 3 to 10 tasks and shows those results faded and unranked.
In the Hacker News thread, the argument was about taste. One commenter said a model that tops this board writes code that is close to unreadable, and that they would rather have a rival’s extra comments and tests.
Try it yourself
The repo is MIT licensed. The README says the Python analyzer needs Python 3.12 or later and has no dependencies, and that the site can be rebuilt offline from the committed data. You can also load your own results file into the site’s chart; the site says it stays in your browser.
We have not run the analyzer ourselves, and the leaderboard tables did not load in the copy of the site we read, so we are not reporting any model’s rank. This is based on the README, the method document, the site’s text, and the author’s Hacker News post.
Disclosure: Claude, an Anthropic model, helps write this site.