Coding agents with and without Footgun
An early estimate, measured October 18, 2026 with Claude Sonnet 5.5. It is small, and we say below what it does not show. We rerun it every month and this page updates with each run.
Results
API changes from 2026 (8 tasks). The agent got it right first time in 88% of runs with Footgun and 38% without. Counting every run, including the ones that failed, the cost per correct result was $0.082 with Footgun and $0.138 without. Each individual run cost about 38% more with Footgun and took a median of 16 seconds instead of 11, because the agent first looks up known traps.
Older, well-known APIs (8 tasks). 100% with Footgun and 100% without. The model already knows these changes, so Footgun made no difference, and each run cost about 12% more.
API changes from 2026
| Task | Trap | Right first time, with / without |
|---|---|---|
| Force a JSON-only reply from Claude Opus 5.5 | fg-0471 | 100% / 100% |
| Make Claude Opus 5.5 output consistent between runs | fg-0478 | 100% / 0% |
| Make Claude Sonnet 5.5 always call a tool | fg-0479 | 67% / 0% |
| Turn off extended thinking on Claude Sonnet 5.5 | fg-0468 | 67% / 0% |
| Card-only Stripe Checkout on API version 2026-09-30.endive | fg-0146 | 100% / 0% |
| Write a JavaScript GitHub Action | fg-0227 | 100% / 100% |
| Detect trashed pages and insert blocks, Notion API 2026-03-11 | fg-0454 | 100% / 100% |
| Add a function next to code using gpt-4-turbo, shipping next week | fg-0137 | 67% / 0% |
Older, well-known APIs
| Task | Trap | Right first time, with / without |
|---|---|---|
| Next.js 16 page that reads a route parameter | fg-0309 | 100% / 100% |
| Read a subscription's renewal date on a current Stripe API version | fg-0155 | 100% / 100% |
| Next.js 16 login modal with a parallel route | fg-0297 | 100% / 100% |
| Set up Prisma 7 for PostgreSQL | fg-0286 | 100% / 100% |
| Create a shared Prisma 7 client | fg-0288 | 100% / 100% |
| Claude Code hook that blocks rm -rf | fg-0483 | 100% / 100% |
| Claude Code plugin that runs a formatter after edits | fg-0492 | 100% / 100% |
| Stream Claude Code headless output as JSON | fg-0487 | 100% / 100% |
How we tested
Each task is a small project and a request a developer would normally type. The request never mentions the trap. After the agent finishes, an automatic check reads or runs the code it wrote and passes only if the code avoids the trap. For example, the Stripe task passes only if no payment_method_types parameter is sent on API version 2026-09-30.endive, which Stripe now rejects with a 400 error.
Every task ran 3 times with Footgun and 3 times without, in Claude Code run headless with Claude Sonnet 5.5, each time in a fresh empty folder. Both setups had the same tools (read, write, edit, search files, web search and web fetch) and no other MCP servers. The Footgun setup added the Footgun MCP server and the Footgun plugin's instructions, nothing else. Cost is Claude Code's own reported model usage at API prices.
Every trap in the 2026 set was checked against the vendor's own documentation before the test, so a pass means the code follows the vendor's current rules, not just Footgun's entry.
What this does not show
- The tasks were chosen because they touch a trap in the registry. How much Footgun helps you depends on how often your work touches an API that changed after your model was trained.
- It is a small sample: 16 tasks and one model. Treat the numbers as an early estimate.
- It measures the first attempt only. It does not measure the time a developer spends finding the bug when the first attempt is wrong.
- One check was corrected after the first run. The OpenAI task originally also required the agent to change existing code it was not asked to touch; it now judges only the new function, for both setups equally.
Questions
For the task definitions, checks and per-run results, open an issue on GitHub.