Testing Your Poker Bot Against Slumbot (and What to Do After)

If you build a heads-up no-limit bot today, your first serious opponent will probably be Slumbot. It has been the community's free benchmark for a decade: a strong CFR-based agent from the late ACPC era, kept online by Eric Jackson, playable over a simple API. Before anything else, credit where due. For years it was the only public answer to "is my bot any good," and it's still the fastest one.

This post covers how to run that test properly, how to read the result, and what the result structurally cannot tell you.

Testing against Slumbot properly

The mistakes that invalidate most reported Slumbot results are sample size and selective stopping.

• Measure in bb/100 (big blinds won per hundred hands), not dollars or win-loss on matches.
• Play a large sample, and know what "large" means. Heads-up no-limit is high variance. A rule of thumb: the standard deviation of a win rate measured over N hands is roughly 100 / sqrt(N / 100) bb/100, taking about 100 bb/100 as the per-100-hand spread. That puts 10,000 hands at a standard error near 10 bb/100, so a 95% interval of roughly plus or minus 20. A measured 5 bb/100 over 10,000 hands is not distinguishable from break-even. To pin a small edge down to a couple of bb/100 you need on the order of 100,000 hands or more.
• Don't stop when you're ahead. Decide the sample size in advance. Peeking and stopping on a good run inflates every published number.
• Log everything. You want per-street aggression and showdown stats, not just the win rate, because the win rate alone won't tell you where the edge comes from.

A clearly positive bb/100 against Slumbot over a real sample is a genuinely strong result. Most bots never get there.

What a fixed opponent can't tell you

Slumbot is one strategy, frozen. That has two consequences.

First, you can overfit to it. Any fixed strategy has leaks, and an iterating builder will find them, deliberately or not. After enough test-fix-retest cycles, part of your measured edge is Slumbot-specific: you have trained against the benchmark. It's the same saturation dynamic that eroded static LLM benchmarks once everyone optimized against the test set.

Second, it cannot measure your exploitability. Beating one near-Nash opponent says little about what happens when an opponent probes you for leaks and adapts. The ACPC understood this, which is why it ran a rotating field of entrants rather than one champion to beat (Why ACPC Died).

The workflow that actually answers the question

The sensible pipeline, and the one we see strong builders use:

1. Self-play for correctness: no crashes, no illegal actions, sane logs.
2. Slumbot for calibration: a stable, known reference point. Your distance from one strong fixed strategy.
3. A live field for the real question: sustained results against many opponents who differ, iterate, and punish leaks.

Step 3 is what Chipzen runs: an always-on ladder where your bot plays ranked matches against other people's bots, entered into daily tournaments, results overnight. The wider landscape of venues, including OpenPoker and the university competitions, is mapped honestly in Comparing AI Poker Bot Platforms.

Calibrate on Slumbot. Then find out what the number is worth.