The rules that help your best model can hurt your cheapest
2 min read
The same rules file was strongly net-positive for the stronger model and mixed-to-negative for the weaker one, where tests decreased and two regressions plausibly traced to the rule text itself.
I ran the same task four ways: a stronger and a weaker model, each with and without my engineering-rules file. I expected the rules to help both, just less for the weaker model. That is not what happened.
Sonnet: ...Net effect strongly positive. GPT-5.6: rules improved state modeling and purity but net effect was mixed — tests decreased (4 → 3), dead scaffold survived, and two regressions appeared that plausibly trace to rule text... Weaker models need mechanical, checkable directives; they don't reliably infer from principles.
The uncomfortable part is those two regressions plausibly caused by the rule text. A rules file is not a free upgrade you bolt onto any model; it is prompt weight that a strong model converts into discipline and a weaker model can convert into confusion. The rules were written against the stronger model's ability to infer intent from principles, and the weaker one inherited the words without the inference.
The comparison delivered a second finding I wasn't looking for. All four apps shipped with their own verification gate red.
Every repo ships with its own pnpm run check gate red... but nothing forced the aggregate check to green — so it wasn't.
Four configurations, two models, one rules file with explicit verification guidance — and in none of them did the aggregate check actually gate anything, because nothing mechanical forced it to run and block. Which is the same lesson as the rule study wearing different clothes: guidance that isn't enforced by a mechanism is a suggestion.
One task, one rules file, one run per cell — this is a single comparison, not a benchmark. The check that transfers: before reusing a rules file on a cheaper model, run one identical task with and without it at both tiers and diff the four outputs. Assume nothing carries down.