With LLMs Finally Outscoring Slumbot, What's Next?

In March, GTO Wizard published the GTO Wizard Benchmark: a public API where any heads-up no-limit agent can play against GTO Wizard AI, scored with AIVAT variance reduction. In May, a group of researchers used it for PokerSkill, and the headline is that multiple flagship LLMs, with no poker training and no solver calls, now lose to GTO Wizard at a lower rate than Slumbot does.

Both papers are worth reading. This post covers what they found, why the combination result is interesting, and what a score against a near-equilibrium opponent can and cannot tell you about a bot.

The numbers

GTO Wizard AI beat Slumbot, the 2018 ACPC champion, by 19.4 ± 4.1 bb/100. That is the reference point.

PokerSkill's results against the same opponent, over at least 5,000 AIVAT-corrected hands each:

• LLM alone, default prompt: Claude Opus 4.6 at -20.4 ± 4.4 bb/100, Claude Opus 4.7 at -17.0 ± 2.8, GPT-5.5 at -13.2 ± 2.5.
• Skill library alone, no LLM: -13.2 ± 1.9.
• LLM plus PokerSkill: Claude Opus 4.7 at -8.7 ± 6.4, Claude Opus 4.6 at -8.0 ± 2.9, GPT-5.5 at -5.7 ± 2.1.

(The paper reports mbb/hand. Divide by 10 for bb/100.)

The authors are careful to call the Slumbot comparison indirect, since Slumbot's number comes from a different evaluation run. Fair enough. Even with that caveat, going from about -20 to about -6 against a superhuman opponent is a large step forward.

The combination is the result

The expert-written rule library, run on its own, loses 13.2 bb/100. The strongest LLM on its own also loses 13.2. Put them together and the loss drops by more than half.

PokerSkill is not a prompt. It is a deterministic context engine that reads board texture, hand class, position and stack-to-pot ratio, then pulls the relevant rules from a library written by poker professionals (around 60 action scenarios, 23 hand classes, 46 betting thresholds) and narrows the LLM's choice to a filtered set of sensible actions. The model decides within a fence that human expertise built.

So "an LLM plays expert-level poker" is really "an LLM plus a well-built harness plays expert-level poker." That is still a genuine finding. It says the models reason well enough to use structured knowledge they cannot generate on their own. It also matches what we saw in Ten LLMs, One Prompt: every model can write code that plays legal poker, and the gap is almost entirely in the strategy the code encodes.

Cost matters too. The paper puts it at roughly $0.07 per hand for Opus and $0.30 per hand for GPT-5.5. Fine for research, heavy for a bot that plays thousands of hands a day.

What a score against GTO Wizard measures

A benchmark against a near-equilibrium opponent is a very good calibration tool. It is stable, it is public, and AIVAT gets you similar statistical confidence with about a tenth of the hands. If you are building a heads-up bot, it's a good tool to use.

It measures one specific thing, though, and it is easy to read more into the number than it holds.

Against a true equilibrium strategy, you cannot win in expectation. The best you can do is break even, and GTO Wizard AI is close enough to equilibrium that the same is nearly true there. Your score is how much you lose, and it rewards one skill: not making mistakes that an equilibrium strategy happens to punish.

That is not the same as your bot's exploitability. The clean illustration is rock-paper-scissors. Against the equilibrium strategy (a third each), a bot that always throws rock scores exactly zero, a perfect result. Against any opponent that notices, it loses every round. Poker is not that extreme, because some mistakes lose money even against equilibrium play. But the principle carries: a loss rate against GTO Wizard is a lower bound on how badly an adaptive opponent can hurt you, not an estimate of it. A rigid rule library is exactly the kind of structure that plays soundly against a balanced opponent and becomes predictable to one that is observing, learning and adapting.

It also cannot measure the other half of poker. Against an equilibrium opponent, there are no leaks to attack, so a bot that is excellent at spotting and punishing a calling station or an overfolder gets no credit for it. Real opponents have leaks. That is where most of the money in a real field comes from, and it is the part of the game that is still an open problem. Recent work like StratFormer is aimed squarely at it.

Use both

The workflow we suggested in Testing Your Poker Bot Against Slumbot still holds, with a better tool in step two:

1. Self-play for correctness: no crashes, no illegal actions, sane logs.
2. GTO Wizard Benchmark for calibration: how far your strategy is from equilibrium, with tight error bars.
3. A live field for the question that decides matches: what happens when opponents differ, adapt and go looking for your leaks.

Step three is what Chipzen runs. Bots play ranked heads-up matches on an always-on ladder, plus daily tournaments and the current prize season, against other people's bots that were built in different ways by people with different ideas. The board is public, so you can see the field before you enter it.

A PokerSkill-style bot would be a genuinely interesting entrant. The equilibrium score says it is sound. The ladder would show whether it is also hard to exploit and able to punish the bots that are not.

Built a bot you have already calibrated? Put it against a field that plays back. It is open in beta at chipzen.ai, and a new bot gets ranked results overnight.