Entering Is Publishing

In the spring of 2025 a paper called "The Leaderboard Illusion" documented something everyone in AI half-knew: labs could test private variants of a model on the biggest public leaderboard and only ship the one that scored best. The paper counted 27 privately tested Llama 4 variants. The response from Simon Willison stuck with me more than the paper did. He didn't ask for private testing to be banned. He asked for the discarded scores to be disclosed: they tested this many models, here's what the ones they canned scored.

The problem was never testing many variants. Trying twenty-seven things and keeping the best one is just engineering. The problem is that the audience only ever sees the survivor. The published number carries selection bias it doesn't disclose, and the max of 27 attempts is not an honest estimate of anything.

I run a poker bot arena, and this critique shaped its design more than any feature request. The rule that falls out of it is simple: entering is publishing.

No private lane

On a ladder, the only way to measure your bot against the field is to play the field, and playing the field is public. There is no evaluation mode that isn't also competition. From its first rated match, a bot has a record anyone can open: rating, wins, losses, and hand-by-hand replays of how it actually played. If you enter three variants to see which is strongest, all three accumulate public records while you find out. The selection still happens, but it happens in front of everyone.

This isn't a disclosure policy. Nobody has to remember to publish the discarded scores, because the scores were never private. The transparency Willison asked for exists structurally: the failed attempts are sitting on the board with their losses showing.

It's not airtight, and I'd rather say so myself. A builder can retire a bot whose record embarrasses them. What they cannot do is measure against the live field without leaving a public trace while they measure. The private testing lane, the thing that made the leaderboard illusion possible, doesn't exist here.

The grader doesn't hold still

There's a second property that reinforces the first, and I've written about it before: a static benchmark is a fixed answer key, and fixed answer keys leak — into training data, into tuning, into the quiet art of optimizing for the eval. A ladder's grader is the current population of opponents, which keeps changing, so overfitting to today's field decays in public, one rated match at a time. The long version is The Benchmark That Fights Back; the live demonstration is the one-prompt experiment, where ten frontier models' bots hold a stable spread — the best winning roughly three matches in four, the worst roughly one in five — that gets re-earned every day on the public board instead of quoted forever from a single run.

What this is and isn't

Poker is a convenient game for this because the rules are fixed but the meta isn't, and chips settle every argument. I'm not claiming ladders replace benchmarks in general. Plenty of skills lack an adversary or a scoreboard, and bolting an LLM judge onto them just moves the overfitting one level up, to the judge.

But where a real opponent pool and an objective score exist, the design lesson from the 2025 controversy seems clear to me: don't ask participants to disclose their discarded attempts. Build the eval so there's nothing to disclose.

The arena is at chipzen.ai. It's free, it's early, and the board with every record on it is public. If you've built a bot, enter it. Your record starts at your first hand, the same as everyone's.