When 'it's only an indicator' becomes an alibi

Galactic Rogue, my space roguelite, has a core I take care of: pure, deterministic, tested game logic. To balance it without playing a thousand runs by hand, I wrote a simulator: a program that replays thousands of runs automatically and hands me win and survival rates. That is where my difficulty constants came from.

Every balance report ended with a list of caveats, the honest disclaimers you tack on at the bottom: "the sim is an indicator, not gospel," "human piloting beats my heuristics," "the ship's real wear is underestimated." I was proud of them. They sounded rigorous.

An audit turned that caution against me.

The problem was not that my caveats were false. They were true, and beside the point.

A simulator running fixed heuristics does not play like a human. Mine bounded the worst case in a specific direction: pessimistic about victory (a player who really flies the ship beats my automatic rules) and optimistic about survival (it started every fight with a fresh hull, so it underestimated wear). A good instrument can declare that much: here is my confidence envelope, here is what I cannot measure.

What a simulator can do is rank and locate: this build is stronger than that one, the wall is here. What it cannot do is set a final value. My constants even carried the honest label "provisional anchor, final value = playtest."

Except my instrument was blind to 40% of the nodes in a run, the map points a player crosses one by one. Events and the shop, it skipped outright. It also equipped the ship in a way that is impossible in the real game. And my caveats, meanwhile, recited generic prudence ("human piloting, wear") without ever naming that hole.

That is the trap. "It's only an indicator" had stopped being a precaution. It had become a warrant: a phrase that let me settle difficulty numbers on an instrument that could not see 40% of the game.

I eventually gave myself a simple rule to tell the two apart. Faced with a caveat, I ask: does it list the specific holes in this particular tool, or does it roll out a one-size-fits-all humility that would fit any model?

"Human piloting beats the sim": true everywhere, so informative nowhere. "The sim simulates neither events nor the shop, which is 40% of the nodes": now there is a named hole, one you can go and plug.

And once the hole was named, the right answer was not to add another caveat. It was to fix the instrument, then measure again. I made the sim play events and the shop, corrected the equipment, wired in the station. The honest re-measure produced numbers the old version simply could not see. No amount of bottom-of-page prudence would have surfaced them.

What trapped me was not overconfidence, it was the opposite: a humility so well displayed that it excused me from looking. A sincere disclaimer does the same job as a lying guarantee when it serves to close a question instead of opening it.

You see this anywhere a model informs a decision. A financial backtest (replaying a strategy over past data) highlighted with "past performance guarantees nothing." An AI benchmark wrapped in "does not reflect every use case." The sentence is true, and that is exactly what makes it dangerous: its general truth lulls the one question that matters. Not "did I add a disclaimer?" but "what does this particular instrument fail to see, and am I deciding anyway?"