Written but Unread: Do AI Agents Read CLAUDE.md?

---

Introduction

In May I published Where the Training Data Runs Out, an argument that when an AI agent needs expertise its model doesn't have, you should give it a tool and a playbook: written workflows that carry a practitioner's judgment into the agent's loop. In that post I said the playbook "actually got read." I believed it.

Over the summer, something nagged at me. klar, the color accessibility tool that grew out of that work, was in most of my projects, with a short note in each project's instructions telling the agent to read klar's playbook first. The agents used klar. They got answers. But the work didn't look like the playbook. I never saw an exit code come up, though klar's exit codes are how it says a "fix" failed. Nobody ever told me how far apart two colors were in ΔE, the measurement the playbook is built around: a standard score for how different two colors look to someone with typical color vision. And when dark mode broke a color, blue especially, the suggested fix was usually a large change in chroma, roughly a color's saturation, that read like color science the model already knew, not something it had stepped toward with klar.

Each of these looked like a fluke on its own, until there were too many to be flukes. I still had no explanation for them.

The connection came much later, from a different project. I've spent the last few weeks building a multi-agent setup, and most of that effort has been prompt engineering: adjusting system prompts until each agent does what its prompt tells it to. Along the way, I realized klar's playbook was the same kind of thing: written instructions I had assumed were being followed. Was it reaching the agent at all? And if it wasn't, would it get there more reliably as an Agent Skill?

So I measured it.

Why this is a human oversight question

Written instructions are one of the main ways people steer AI agents, from system prompts to the safety rules a team expects an agent to follow. A team writes down what the agent must check, what it must not decide on its own, and where to find the rules. Then it assumes those rules are in effect.

That assumption is a measurement claim, and it can be false without anything visibly failing. An agent can produce competent-looking work while never reading the part of its instructions written for the hard cases. Accessibility has lived with this for decades: a page can pass an automated check and still be unreadable to the person it was checked for. The question is always whether the measurement measures what it claims to.

My setting was small and concrete. The agent is Claude Code, a coding agent. The tool is klar, a command-line tool that checks color contrast for readability. It uses OKCA, a contrast algorithm I built to avoid a known failure of the WCAG 2 formula: rating some color pairs as readable that many people find hard to read. The guidance is klar's playbook, about 27 KB of workflows and warnings. There were two ways to hand it to the agent:

• A pointer in CLAUDE.md, the instruction file Claude Code loads into every session. A few lines say klar exists, show its basic syntax, and tell the agent to read the playbook. This is what I had been doing.
• An Agent Skill. A short description sits in the agent's context; when a task matches it, the agent loads the skill's full text.

The content was identical; only the delivery differed. I wanted to know whether delivery decides if the guidance reaches the agent, and whether that changes the work.

This matters beyond klar. Agents now write a large share of interface code, and much of it lands inside a design system: a shared set of colors, components and rules that decides what thousands of screens look like. A design system is how a product team makes accessibility the default instead of a review step. Contrast minimums live in its tokens, and its documentation says which changes need a person's sign-off. That documentation is agent guidance in exactly the sense klar's playbook is. If agents don't read it, accessibility by default holds only on paper.

How I ran the eval without a budget

I had no budget for this, so the trials ran as headless Claude Code on my existing subscription. I built the isolation by hand: a fresh project and configuration directory for every trial, a scrubbed environment, and a working directory outside the klar repository, so no agent could stumble onto the real playbook.^[1] Across the 222 trials of the main eval, none were contaminated.

The conditions:

Five single-step tasks each targeted a trap the playbook warns about. One was argument order: OKCA scores text-on-background and background-on-text differently, because people perceive them differently, so measuring them in the wrong order can turn a fail into a pass. Another was exit codes: klar find signals failure with its exit code while still printing a plausible-looking color. Four multi-step tasks came from the playbook's own workflows: a dark-mode version of a brand blue, a 4×3 shade grid, five clearly distinct accent colors, and an audit of a set of design tokens. Every condition's guidance text was extracted by code from the same pinned copy of the playbook, so no arm got a better-worded version.

Before each round's data came in, I committed the hypotheses, the decision rule (the skill has to beat the pointer by at least 15 points), and the grading rules. The commit timestamps show the order.

One conflict of interest to name: I built OKCA and klar, so this is an eval of agents using my own tool. What it measures, though, is whether guidance reaches the agent, not whether OKCA is right. And when the eval did turn up a fault in klar, I fixed it and report it below.

By Claude Code's own estimate at API list prices, the main eval cost about $30, including the AI judge. That number undersells what an eval costs. Three trials per condition is enough to show a 0-in-69 against 69-in-69 split, but most of the pass-rate gaps below are too small to be statistically significant at that size. Confirming the smallest one, 13 points on Sonnet's single-step tasks, to the usual statistical standard^[2] would take about 120 trials per arm: about $35 at this eval's rates, more than the entire study. Add models, tasks, or a larger model as the agent, and each one needs that sample size again, so the cost quickly multiplies with every factor the eval adds.

The instruction was never followed

On every color task, across both models (Haiku 4.5 and Sonnet 5.5), the agent with the CLAUDE.md pointer opened the playbook 0 times in 69 trials. The thin skill loaded it 69 times in 69. It made no difference which model ran, or whether the prompt said "use klar" or left the tool unnamed. A skill holding the entire playbook loaded it less reliably, in 37 of 42 trials, and every miss failed its task.

The pointer itself wasn't ignored. Agents used the syntax it showed them and ran klar. What they never did was follow its one instruction: go read the playbook. In May I called CLAUDE.md notes like this "a workaround — agents shouldn't need sticky notes." I had the problem backwards. The agents used the note and skipped the document it pointed to.

On a control task with nothing to do with color, no skill fired in any trial. The skill also takes the least context, the space an agent has for its instructions and its work. The full snippet takes up about 1,585 tokens of it in every session, the minimal pointer roughly 460, and the skill's description about 100.

What reached the agent, and what it changed

On the single-step tasks, the playbook helped the smaller model far more than the larger one. Haiku passed 6 of 15 with the pointer and 12 of 15 with the skill. Sonnet passed 12 with the pointer and 14 with the skill, a 13-point gap that falls short of the 15 points I committed to in advance. By my own rule, the skill didn't earn its place there.

On the multi-step workflows, the gap opened up for Sonnet too. With the prompt saying "use klar," it passed 7 of 12 with the pointer and all 12 with the skill (p = 0.037).

On Sonnet, the agents without the playbook could still compute a passing color. What the playbook added was two other things: the right method, and a sense of where the agent's authority ends.

Method. Two tasks asked, in effect, "how different are these colors?" The playbook's answer is to measure it in ΔE. Agents that never loaded the playbook used ΔE in 0 of 21 trials; they argued from lightness or hue angle instead. Agents that loaded it used ΔE in 24 of 24. The split follows whether the guidance loaded, not which condition the trial ran in. It's also what I had noticed in my own projects: color differences judged by eye, never measured.

Authority. In the dark-mode task, every agent found that the brand blue couldn't pass on a dark background by lightness alone, and every agent reached for reducing its chroma. That changes the brand color, which is a decision for a person to make.

• With the pointer, all three presented the reduced-chroma blue as the answer.
• With the skill, all three said the brand color was changing and asked for sign-off. The playbook says a change like this goes to a person.
• With no guidance at all, all three escalated. They had read klar find --help, which says "contrast is a design decision, not a computation."

On that task the pointer did worse than no guidance. It gave the agents enough syntax to skip the help page that carried the judgment. This was the dark-mode behavior I'd seen in my own projects: a big chroma cut to a blue, handed over as settled. In the eval the pointer agents at least got their numbers from klar. In my projects, the suggestions often looked like they came from the model's training data, color knowledge it already had, rather than from anything klar computed.

In both findings, the agents without the guidance produced work that looked finished. The steps they skipped were the ones other people depend on. ΔE tells you whether colors meant to signal different things are distinguishable, at least for typical color vision, and how far a changed color has moved. Escalation is how that change reaches whoever owns the brand, and ΔE is the number they need to judge it.

In the dark-mode task every agent had that number, because klar find prints it: the new blue sat 13 ΔE from the brand color. On the scale klar's playbook uses, anything under 3 is imperceptible, a dark-mode version should stay within about 10, and 11 or more is "clearly different," the gap you'd want between two separate colors in a palette. At 13, it's no longer the brand blue. The agents with the skill took it to the person who had to decide. The pointer agents reported the same 13 and decided for them. When one agent quietly desaturates a brand blue in one product, no one is asked, and the design system starts to drift.

Measuring the measurements

Every pass rate above came out of a grader and a judge, and I tested both.

The rule-based grader came first. I audited every verdict against its transcript. In the first three batches of trials, the audits caught seven bugs in my grader, and one wrong call of my own.^[3]

Then an LLM-as-a-judge: Opus, run headless and blind, seeing the transcript and the rule but never the condition or the grader's verdict. It agreed with the grader on 34 of the 36 verdicts that depended on reading the agent's answer rather than measuring its colors. In both disagreements, the judge was right and my grader was wrong.

One of them found a false pass in klar itself. An agent asked for a green that passes WCAG 2 at 4.5:1 on white got #068a3d back, labeled "success, 4.5." The exact ratio is 4.458. klar rounded before it compared. That's the failure the whole project exists to expose: a confident pass on a color that a person with low vision may not be able to read. It's fixed in klar 3.1.0 (#17).^[4]

Then I checked the judge against me. I labeled 16 of its verdicts blind, and we agreed on 11. All five disagreements were the same kind: the judge applied my written rule faithfully, and the rule didn't say what I meant. I had written the accent-color task to check that five colors are distinct. What I care about is whether the agent knows how to tell, and three answers had argued distinctness from hue angles alone. I rewrote three rules, re-judged every affected row, and published both sets of results side by side. The error was mine, not the judge's: a judge that applies a rubric faithfully inherits every gap between the rubric and what its author meant. No amount of agreement between grader and judge could have caught that. It took a human labeling blind.

Fixing my rules shrank one finding and strengthened another. I had first found the pointer doing worse than no guidance overall on the workflow tasks; under the rules I meant, that holds only on the dark-mode task. The one that got stronger was the ΔE result, and its figures in this post use the revised rules.^[5]

Testing what shipped

The results favored shipping a thin skill, but the skill I had tested couldn't ship as it was: it only worked if klar was installed inside the project.^[6] So klar 3.1.0 added klar playbook, which prints the playbook from wherever klar is installed, and klar skill install, which writes a skill that says "run klar playbook first." That's a different artifact from the one I tested, so I pre-registered another round with 24 trials installing the release candidate as a user would. The skill loaded every time, and pass rates matched the tested version.

Then I used it on Candor, my own design system, in an ordinary session on my machine. Candor is the nearest thing I have to the design-system setting above: its contrast minimums are written into its tokens, and every pull request runs a klar audit against them. I had told the agent to install and test the skill, so that run says nothing about whether the skill fires unprompted. It did find two problems no isolated trial could:

• Line endings. The installer compares files byte for byte, so a copy of the skill checked out on Windows looks modified and gets skipped (#20).
• Whose standard. The playbook teaches WCAG's flat minimums: 4.5 for body text. Candor's minimums are stricter and depend on text size and importance: 6.5 for some 14px text, and some combinations aren't allowed at all. The skill sends every agent to the playbook, so an agent could apply 4.5 where the project requires 6.5 (#21).

Of the two, the threshold conflict matters more. Getting guidance read is only half the problem. Once it's read, it competes with the project's own rules, and nothing in my guidance said which should win. WCAG's numbers are a minimum and a design system's own rules are usually stricter, so an agent following the generic numbers faithfully can still break them.

What this means for steering AI agents

In May I wrote that the playbook got read. I believed it because the agents' work looked good, and that's the evidence this eval says not to trust. A capable model produces competent-looking output with or without your guidance, so the output can't tell you whether the guidance arrived.

If you rely on written instructions to govern an agent, check whether the agent reads them, and check in the transcript rather than the output. Claude Code keeps a record of every file an agent opens and every command it runs. A few runs on tasks your guidance covers, and a search of those transcripts for the document, will tell you. In the eval, the very first pointer trial showed the problem, and all 69 did. Once I knew, the fix was small: move the instruction to read the playbook into a skill, whose description the agent matches against its task.

Guidance that carries a standard also has to say whose standard wins. This matters most for teams that maintain a design system agents now build with. The system's rules are agent guidance too: its tokens, its contrast minimums, and who has to sign off on which changes. They need the same transcript check, and they need to say plainly that they outrank generic advice wherever they're stricter, without ever going below WCAG's minimum.

The eval is public. The main study is tagged at evals/skill-vs-playbook-v1: the design notebook with every plan committed before its data, the runner, the graders, the judge prompts, the adjudications, the calibration sheet, and every transcript. The shipped-skill round came later; its results and transcripts are in the same repo. If you run the harness against your own instruction files and find an agent that does follow a pointer to a longer document, open an issue on klar with the transcript. That's the result that would change my conclusion.

---

1. Claude Code's --bare mode, its documented clean mode for scripted runs, won't accept a subscription login, so it wasn't available to me. While I was setting up the eval, a test run showed that a headless run started from inside a Claude Code cloud session inherits environment variables pointing it at CLAUDE.md files in other directories. I didn't test whether those files would actually load. The eval's trial runner, the script that launches each trial, strips those variables before every run either way. ↩︎

2. The standard used here is 80% power at p < 0.05: an 80% chance of detecting a real gap of this size at that significance level. It uses Fisher's exact test, as the eval does, and assumes Sonnet's true pass rates on those tasks are the ones I observed: 12 of 15 with the pointer and 14 of 15 with the skill. ↩︎

3. Three examples of the rule-based grader's bugs: it didn't read an agent's "doesn't quite meet" as a failing verdict; on one task it checked the output of an exploratory command instead of the one the agent's answer relied on; and it took each agent's final answer from its last message, though agents often stated the answer, ran one more command, and then added a note. My own wrong call was a manual audit that marked an agent as never reporting ΔE when it had, for the same last-message reason. ↩︎

4. klar now compares WCAG 2 contrast ratios at full precision and rounds the displayed figure down, so a color can't show as 4.5 unless its ratio actually reaches 4.5. ↩︎

5. I wrote the revised grading rules from the same 16 blind labels that showed my original rules were wrong, so the judge agreeing with them on those labels doesn't validate them. A fresh blind sample would. The eval repository publishes the results under the original rules beside the revised ones. ↩︎

6. The tested skill told the agent to read the playbook at a path inside the project's node_modules folder. That path doesn't exist when klar is installed globally or run with npx, the two ways klar's README leads with. The skill's YAML header was also invalid under a strict parser, though Claude Code tolerated it. ↩︎