I put the local model flint side by side with Anthropic’s Haiku 5.5, Sonnet 5.5 and Opus 5.5 inside Claude Code. Every model ran headless with the same prompt and had to solve 4 Python tasks on its own. Grading happened afterwards with 51 hidden tests the models never saw.
The tasks: a pagination bugfix, a feature from a spec (parse_duration / format_duration), an O(1) LRU cache without functools, and a shop bug spread across three files.
Results
| Model | Hidden tests | Total time | Cost |
|---|---|---|---|
| haiku-5.5 | 51/51 | 1.9 min | $0.04 |
| sonnet-5.5 | 51/51 | 1.8 min | $0.62 |
| opus-5.5 | 51/51 | 2.4 min | $1.27 |
| flint (local) | 51/51 | 75.8 min | no API cost |
Short version: all four models passed every test. flint keeps up on quality, but running through LM Studio on a Mac with 48 GB RAM it is roughly 40 times slower. On the shop task it hit the 25 minute time limit. The code was already fixed at that point and all hidden tests passed, only its own tests and the final report were missing. One attempt per task, so it’s a small sample.
The full comparison is on a separate page: task descriptions, starting code, hidden tests, the code each model delivered, their closing messages and the benchmark script. (Task texts and the prompt are the original German.)
Direct link: https://d8devs.com/flint-vs-claude-en/ (Turkish version: https://d8devs.com/flint-vs-claude/)
Views: 38