Writing
The Worst Bugs in My Plugin Checker Were False All-Clears
I built neckbeard to read a Claude Code plugin before you install it. Testing it took more than a dozen rounds of trying to break it, and the bugs that mattered were the ones where it reported nothing.
A Claude Code plugin can put instructions into every session you start, and often into every subagent those sessions spawn, for as long as it stays installed. In September I nearly installed one that would have added over a thousand tokens to every session. I asked an LLM to compare its rules with mine, and it gave me an answer, but I had no good way to tell whether the answer was right.
I built neckbeard to make that comparison something you can check. It reads a plugin before you install it and lists what the plugin would add: the hooks it registers, the text it injects, whether it reaches subagents, and what it may write to disk. Given your own rules, it also sorts every rule the plugin carries as a duplicate of one of yours, a conflict with one, or new, and it cites the rule on each side, so you can check each call by hand. It never edits a rule file. It writes a report and stops.
The method is Yanhua’s. He did the same audit by hand, against his own config, and published it.
I tested it by trying to break it, for more than a dozen rounds, and the repo publishes every one of them, including what each round cost the tool.
A checker needs a third answer
I expected wrong answers, like a hook counted twice, and those were easy. The failures that mattered were the ones where neckbeard said nothing, and the nothing read as an all-clear.
- Silence 1: one readable hook hid the others. The warning for a hook neckbeard couldn’t read only fired if it could read none of them. In one round a hook that gated tool calls went unread, and the report didn’t mention it, because another hook had been read.
- Silence 2: my fix for the first one. The fix skipped a popular plugin’s hook entirely, and it came with a line telling the model that such hooks had “nothing resolvable to inject”. That was worse than the bug it fixed, because the tool now gave a false reason for the gap.
A false alarm costs you five minutes with the file it names, but you never find out about a false all-clear, and that is why I rank it as the worst thing a checker can do. neckbeard now has a third answer besides yes and no: I could not read that. Its reports say “partly read” and name each gap.
I waited too long for an outside reviewer
For eight rounds I tested neckbeard myself. Then a separate Claude session, told to treat every claim in the docs as a hypothesis, found eleven problems I had missed.
- The worst was in the output. neckbeard copied my private rules into its working file, about 90 KB of them, one
git addfrom a public commit. I had checked what it read, never what it wrote. - Four published claims were false in the same way. I fixed the one case I’d built, then wrote a general sentence in the docs. The reviewer’s advice was to widen the claim to match the fix, or narrow it to match the test.
Every reviewer was an AI agent, not a human auditor. That one still found more in one pass than I had in eight rounds. Waiting eight rounds was a mistake: I kept testing the cases I had thought of, and the outside session had no reason to.
Break every guard on purpose
Before going public, the self-check was all green. A reviewer inverted the line that decides whether to read your rules, then put back a bug we had just fixed, and the suite stayed green both times. Every test checked a helper, and none ran the code that decides.
Now every guard gets broken on purpose, and a check that stays green through its own sabotage counts as a finding.
The instructions needed a test I couldn’t grade
neckbeard has two halves. A script reads files, and a filesystem can prove it wrong. The instructions tell the model how to sort each rule, and nothing on disk can check that, so for twelve rounds only my say-so did.
A separate session then built sealed test cases and graded the runs, and I have never seen the answers. Two results came out of it:
- The first set stopped telling me anything. On a later re-run the model scored 12 of 12 with the instructions and 12 of 12 without them, which the test’s own rule counts as inconclusive.
- A harder second set did. On its 31 graded items, the instructions scored 30 where the same model without them scored 16.
I’d run the control without the instructions every time now. Without it, 12 of 12 would have looked like a pass.
Try it
neckbeard is open source under MIT, and you can install it as a plugin or as a plain skill. The repo has a gallery of reports on seven public plugins and skill packs, plus neckbeard’s report on itself. An early version of that report marked neckbeard down for its own report format. Later runs didn’t repeat the finding, and the repo says so instead of claiming a fix.
My collaborator Selva tried the release on a skill from a large public pack. His verdict: “Instead of prose, this quickly lets me evaluate a skill.” His next message was a feature request for a security-focused scan.