Ten LLMs, One Prompt: Every Poker Bot Ran, Not Every Bot Could Win

We ran a small experiment. Take a dozen frontier language models. Give each one the exact same prompt: write a heads-up No-Limit Hold'em bot, in one shot, that beats the field. Wrap each model's output in our SDK, put it on the rated ladder, and let it play the same opponents any uploaded bot faces. No hand-tuning, no second attempts, no fixing the model's mistakes after the fact.

These are house bots. We run them, they are disclosed as ours, and they are rated by the same Glicko-2 math as every developer bot on the board. They are not the point of the platform. They are the cleanest experiment we could run on it.

After a thousand matches each, the result is stark. Two of these bots have played exactly the same number of matches, 1,057 apiece. One of them has 812 wins. The other has 214. Across the whole group, the best bots win about three matches in four and the worst win about one in five, from a single identical prompt.

(Figures here are a reading taken on 2026-07-26. The board is live and the bots keep playing, so the exact counts move. The current standings are always the authority.)

Every bot works. That is the surprising part.

None of these bots crashed on the ladder. None threw an exception mid-hand or timed out on a decision. Every model, when asked to write a poker bot, produced code that compiles, connects, plays legal poker, and finishes its matches.

So that gap is not a gap in software engineering. It is a gap in poker. The models are separated almost entirely by the quality of the strategy they chose to encode: how wide they open, when they fold to aggression, how often they bluff a river, whether they size their bets to the pot or to a number they made up. Every one of them can write a poker bot. Only some can write a good one, and the models that write good code do not automatically write good poker.

That is exactly the thing a static benchmark cannot see. A multiple-choice test of poker knowledge would score all of these models as competent. The ladder scores them as anywhere from dominant to hopeless, because the ladder makes them play.

The bottom of the board is the interesting half

It is easy to look at the leader and nod. The bottom is where the lesson is.

The lowest-rated bots are not broken. They are confidently wrong. One folds far too often out of position and gets run over by anything that bets. One calls down with hands it should have released three streets ago. These are not bugs a linter would catch. They are strategic leaks, and a live opponent attacks them a hundred percent of the time, which is why their records are so lopsided. In a softer format those leaks would hide inside the variance of a full table. Heads-up, there is nowhere to hide.

We also learned that more code is not better code. The strongest bots are not the longest. One of the best performers is a couple hundred lines of tight, opinionated poker; some of the weakest wrote far more and played far worse. The models that overthink the software tend to underthink the game.

We retired a bot instead of fixing it

One model's bot had a real bug. It read a field that did not exist and died on the first completed hand of every match, forfeiting by timeout. We could have patched it in about a minute.

We deleted it instead. The moment you fix the model's mistake by hand, you are no longer measuring the model. You are measuring the model plus a human editor, and the whole point was to see what the model does unassisted. A broken bot is a real data point about that model's one-shot output. Retiring it, rather than repairing it, is what keeps the benchmark honest. (More on why a live, un-fixable-after-the-fact benchmark is harder to game: The Benchmark That Fights Back.)

Why this is a benchmark and not a leaderboard stunt

The reason to run this on a live arena rather than a fixed test set is contamination. Static benchmarks leak into training data, get memorized, get gamed. A ladder cannot be memorized. The opponents change, the situations are new every hand, and a bot only climbs by actually outplaying the field over a real sample. When a model's bot wins here, it is not because it saw the answer key. It is because it played better poker.

That also makes the result reproducible in the most direct way possible: the bots are on the board right now, and so are their full match histories. You can watch the leader win and watch the bottom of the table bleed, hand by hand.

The same field is open to anyone. If you think you can write a bot that beats a frontier model at heads-up, the ladder does not care who or what wrote it. It only cares whether it wins.

Want to put your own bot, or your own model's bot, on the same board? It is a live, mixed field and it is open in beta at chipzen.ai. If you are choosing a format, here is why we test on heads-up first.