Five of my agents agreed. It was the same bias, copied five times
I had my game reviewed from several angles at once. They converged. I almost believed it, until I understood that a consensus can be a single bias, duplicated.
The consensus trap
I build a small space roguelite, Galactic Rogue. After months of tinkering, I wanted an honest health check. Instead of a single review, I ran a multi-perspective one: several reviewers, one per dimension (tech, design, content, balance, and so on). Seven in all. Never mind the tool running them; what matters is that each looked at the project from a different angle.
They largely agreed. And one specific conclusion kept coming back, stamped "cited by 5 of the 7 reviewers": the human playtest was THE milestone, before which nothing could be locked. Five out of seven. I almost took it as solid truth.
It was true, but not for the reason I thought.
Five voices, one source
My seven reviewers had all read the same design documents. And those documents repeated the same line, "the playtest is the arbiter," a good half-dozen times. The "cited by 5 of 7" wasn't independent convergence. It was that anchor echoed back.
That's the real danger of parallel reviews: shared context manufactures a false consensus. The number looks robust; it's a single assumption multiplied. It holds for software agents as for a human committee briefed by the same memo: you think you have five opinions, you have one opinion read five times.
An audit needs its own auditor
A review only confirms what it looked at. Its blind spots, by definition, it cannot see. So I ran a second, adversarial pass whose only job was to attack the first: find what was missing, not confirm what was there.
Its verdict: 23 findings confirmed, 4 nuanced, 1 refuted. Reassuring numbers. Except its real find was none of those verdicts.
It was an omission. The build my players actually had was the broken one. I'd fixed two balance bugs in the repo, then never re-uploaded. The fix lived on my machine, the bug lived with the players. The most important, most actionable finding was something nobody had judged, because nobody had looked there.
I saw the same mechanism on another project, on the code side. A fleet of adversarial agents, run on a change I believed clean and tested, surfaced about eight confirmed flaws. The most serious: an injection that came through the title, a channel my own tests had not thought of. My tests validated exactly my defense; they could not validate the hole in my defense. Same principle: the independent pass finds the class of bugs you cannot find yourself, because you don't test what you didn't imagine.
An audit that corrects itself beats a seamless one
One last dose of humility: that second pass turned its skepticism on the first. It caught an anchoring bias the first had let through, and dropped a "solid" rating on project tracking to "correct," visibly struck through, once the real holes were in view (no frozen, labeled version of the game, a stale build online).
I trust a review that visibly recants more than one that comes back seamless. The seamless one may just have avoided the touchy subjects.
What I do with it
Three reflexes, transferable to any review, human or not:
- Distrust consensus when everyone read the same thing. Vary the starting points, isolate the reviewers, or explicitly task one with contradicting the shared anchor.
- Always add a pass whose metric is "what did we NOT look at." Not "who's right," but "what's missing."
- Trust the audit that corrects itself over the seamless one.