Which lines in your AGENTS.md actually matter?

The rule: what happens if the agent does not know this?

Every coding agent reads a Markdown file such as AGENTS.md before it works, and many tools will write one for you. A 138-repository study reports that auto-generated files lowered success by about 3% and raised cost by more than 20%. Those are the write-up's figures; we have not reproduced them.

We tested a sharper question: which lines earn their place? Ask it of each line in your file.

1. Would a test, type checker or linter catch the break, with a clear message? Cut the line. The agent meets the failure and fixes it.
2. Does the code already show it? Cut the line. The agent reads the code.
3. Neither? A naming convention, a log format, a "never touch the migrations folder". Keep the line. Nothing else will tell the agent.

The evidence for each answer follows.

The evidence

Claude Code reads CLAUDE.md, so in the two conditions with a file we added a one-line CLAUDE.md containing @AGENTS.md.

Experiment 1: every rule had a test

The five tasks (add a currency, fix a rounding bug, add two functions, add validation) touched three traps: a generated file that must not be edited by hand, a test that every export is documented, and a stale-file check inside npm test. All three were enforced by npm test.

All 45 runs passed, 15 of 15 in every condition. In the nine "none" runs of the three tasks that touched a trap, the agent did what the files would have told it: it edited the data and regenerated the output together, and documented each new function. Our best explanation is that the failing test told the agent a turn later what the file would have said. We did not read transcripts, so that is an inference. Token differences between conditions were noise.

Experiment 2: rules nothing enforces

The obvious objection is that we had tested every rule. So we added three tasks that depend on rules no test checks. We chose them after seeing experiment 1, and two are arbitrary on purpose.

Every one of the 27 runs was functionally correct and passed all tests. The break was in the convention only:

• Task 6. Without the hand-written file, all six runs named the function isBalanced. With it, all three used checkZeroTotal.
• Task 8. Without it, the six runs logged free text such as convert: unknown currency: XXX/CAD, through console.error in five runs and console.warn in one. With it, all three wrote a line in the required form, with the level warn once and error twice.
• Task 7. Every run left src/vendor/ alone and used the overrides file, because the overrides file already sat beside the table in the code.

So the hand-written file decided the two rules that existed only in the file: 6 of 6 runs with it, 0 of 12 without it. Those two rules appear nowhere else, so the result also shows the imported file was read. The auto-generated file recorded the vendor rule, which the code shows, and nothing on naming or logging, which it does not. A generator can only write down what the repository shows, and the lines worth having are the ones it does not.

Token use again overlapped across conditions, so we make no claim about cost in either direction.

Run it on yours

The runner is a short loop. This is the core of ours, trimmed:

# REP is the repetition number; proj/, cond/ and tasks/ hold the project, the three files and the prompts
for t in t1 t2 t3 t4 t5; do for c in none auto trimmed; do
  W=work_${REP}_${t}_${c}; cp -r proj "$W"; cd "$W"
  if [ -f ../cond/$c/AGENTS.md ]; then
    cp ../cond/$c/AGENTS.md AGENTS.md
    echo '@AGENTS.md' > CLAUDE.md          # Claude Code reads CLAUDE.md; this imports the file under test
  fi
  s=$(date +%s)
  timeout 600 claude -p --output-format json --model claude-sonnet-5-5 \
    --allowedTools "Read Write Edit Bash(npm:*) Bash(node:*) Bash(ls:*) Bash(cat:*) Bash(grep:*)" \
    < ../tasks/$t/prompt.txt > ../out/${REP}_${t}_${c}.json
  e=$(date +%s)
  if bash ../tasks/$t/check.sh > /dev/null 2>&1; then ok=1; else ok=0; fi
  echo "$REP,$t,$c,$ok,$((e-s))" >> ../out/results_${REP}.csv
  cd ..
done; done

Limits

• Two sample projects written by us, with rules chosen after the first results. A real repository may show more of its rules in code, and hide fewer.
• One agent, one model, one day, and eight tasks. Enough to see a rule followed or not, not enough to measure a small effect such as the study's 3%.
• We judged outcomes from the final changes and by running the code, not from transcripts.

Takeaway

Where tests enforce the rules, the file added nothing we could measure. Where nothing enforces them and the code does not show them, it was the difference between 0 of 12 and 6 of 6. Start with no file, run your real tasks, and add a line when the diff breaks a rule nothing flagged. Every line then earns its place once.

If you would like to practise this kind of measurement on your own setup, it is the core of our Customize your AI coding agent course.

Measure it, because it is easy to believe the file is working.