Reinforcement Learning for Board Games: A Primer

Most introductions to reinforcement learning start with abstract Markov decision processes and grid-world toy problems. That’s mathematically clean but it skips the historical reason RL became famous: board games. Backgammon, chess, Go — these were where the algorithms first proved they could compete with the best human players. This article is an RL primer specifically through that lens.

We’ll cover the basic ingredients (states, actions, rewards, policies, value functions), why dice games map onto them especially well, and why backgammon was the watershed that made the field mainstream. Every concept is grounded in a concrete board-game example — backgammon for short narratives, long narde for the variant-specific bits.

What RL is, in one paragraph

Reinforcement learning is the framework where an agent learns to act by interacting with an environment and observing the consequences. There’s no labelled training data telling the agent «in this state, the right move is X.» Instead the agent receives a reward (a number) at the end of each episode (game), and over time it adjusts its behaviour to maximise expected total reward.

For a board game: the agent sees the position, chooses a legal move, the dice are rolled, the opponent responds, the cycle continues until someone wins. The agent’s only feedback is the game outcome — +1 for a win, -1 for a loss, possibly intermediate rewards for intermediate goals. From thousands or millions of games like this, RL extracts a policy (which move to play in which position) without ever being told what the right move was in any specific situation.

The five ingredients

Every RL formulation has the same five components. Mapping them onto a board game makes the abstraction concrete:

State (s). The complete information needed to decide what to do next. For backgammon: the position of all 30 checkers (15 per side), whose turn it is, the dice, the doubling cube state, and the match score. For long narde: same plus phase indicator (blocking / race / X-escaped / O-escaped) — though phase can be inferred from checker positions, so it’s redundant in pure state representation.

Action (a). What the agent can do from the current state. For checker plays: the set of legal move sequences given the dice. For cube actions: double / no-double, take / drop. The action space is large for backgammon (typically 5–60 legal play sequences for a roll, depending on position and dice) — much larger than chess’s per-move branching.

Reward (r). The signal the agent maximises. Standard for board games: 0 during the game, +1 at terminal-win, -1 at terminal-loss. Variants can be richer: +2 for a gammon, +3 for a backgammon, with the doubling cube as a multiplier. Long narde uses 1/2/3 for normal/mars/koks. The choice of reward function is consequential — agents trained with single-win-only rewards will not learn to play for gammons.

Policy (π). The function that maps state to action probabilities. The thing the agent is trying to learn. Initially random; over training, biased toward better moves. Many RL algorithms produce a deterministic policy (always pick the action with highest expected value); others produce stochastic policies (sample from the action distribution).

Value function (V or Q). An estimate of how good a state is — the expected total future reward from that state under the current policy. V(s) is the value of being in state s. Q(s, a) is the value of taking action a from state s. Most RL algorithms work by improving the value function and using it to derive a better policy.

Why dice games are good RL targets

Backgammon and narde have three properties that make them especially friendly to reinforcement learning:

  1. Perfect information. Both players see the full board, so state can be captured cleanly without information-set machinery.
  2. Built-in stochasticity. The dice provide natural exploration. An RL agent that always picks the highest-value action would rarely explore alternatives in deterministic games like chess; in dice games, the same position never recurs identically because the dice differ.
  3. Bounded episode length. Most backgammon games end within 50–80 plies; narde similarly. Compared to Go’s 200+ plies or no-limit poker’s variable horizons, this means more training games per unit of compute.

The flip side: dice make the value function inherently noisy. The same move from the same position has different expected outcomes depending on what dice come up next. RL algorithms have to estimate value as an expectation over dice rolls, which adds variance to the learning signal.

TD learning and why TD-Gammon mattered

The most influential algorithm in board-game RL is temporal-difference (TD) learning. The idea is simple: at each step, compare your current value estimate to your value estimate one step ahead (after taking an action and seeing the next state). The difference — the TD error — is your training signal. You nudge V(s) toward V(s’) by a small amount.

In equation form (TD(0) update):

V(s_t) ← V(s_t) + α [ r_{t+1} + γ V(s_{t+1}) − V(s_t) ]

α is the learning rate, γ is the discount factor. The whole update is local — you don’t need the final outcome of the game to start learning, just the next state and reward.

Tesauro’s TD-Gammon (1992) put a 3-layer neural network in place of a tabular V(s) and trained it with TD(λ) using almost-pure self-play. Within a few months of training (on hardware that’s now laughably slow), it played at world-championship level. Two things were unprecedented:

  • No expert games. TD-Gammon learned from self-play, not from a database of master-level games. Compare to chess and Go programs of the era, which relied heavily on opening books and grandmaster databases.
  • The networks discovered concepts. TD-Gammon found priming, blitzing, and back-game strategies that human players already knew, and a few they didn’t. Its preferences influenced backgammon theory.

Modern engines (gnubg, eXtreme Gammon, Nardex) are direct descendants. The architecture is bigger and the training regimes are more sophisticated, but the central idea — TD learning over a neural value function trained from self-play — is the same line.

Action selection: from values to moves

A trained value function V(s) doesn’t directly tell you what move to play. To choose a move, you typically:

  1. Enumerate all legal move sequences given the current dice.
  2. For each sequence, simulate the resulting position s’.
  3. Evaluate V(s’) under the trained network.
  4. Pick the move that minimises V(s’) from the opponent’s perspective (since we made our move and now it’s their turn — we want to leave them with the worst position).

This is just one-ply search. Engines like gnubg-2-ply do another step: for each candidate of yours, they roll all 21 dice combinations for the opponent and compute V over the opponent’s best response, averaging. This gives a better estimate but is 21× more expensive.

Q-learning would store V at the (state, action) level, skipping the explicit search — but for backgammon this is impractical because the action space is too large. State-value V(s) plus search at decision time is the standard.

Self-play as a training engine

Self-play is the training method that made TD-Gammon famous and that most modern systems still use. The setup:

  1. Initialise the value network randomly.
  2. Play games where both sides use the current network for action selection (with some exploration noise).
  3. After each game, run the TD update over the trajectory.
  4. Repeat for millions of games.

The agent doesn’t need any external opponent. It improves by playing against marginally-improved versions of itself. The risk is converging to a local optimum (a strategy that’s stable against itself but exploitable by a different opponent), but for backgammon and narde, self-play has consistently produced strong agents.

A common refinement is to maintain a history of past versions and play against a sampled mixture, not just the current one. This prevents the agent from forgetting positions it had previously learned to handle.

What gets learned, in plain English

Once a backgammon TD-network is trained, what does it «know»? The clearest test is to look at its preferences in well-understood positions:

  • Opening moves. A trained network learns the standard opening ranking (24/13 split for 6-1, 13/11 7/2 for 5-2, etc.) without ever being told these were good moves.
  • Priming. It learns that holding 6-prime-style structures is valuable, even though no one defined «prime» in the input encoding.
  • Pip racing. It learns to count pips implicitly — when contact is broken, it consistently prefers the higher-pip-count side’s positions.

For long narde, similar emergent learning happens around the head rule, the 6-in-a-row prohibition, and mars/koks scoring. We’ve seen networks learn that the 6-prime in narde must be «escapable» before the structural value emerges — they learn the rule’s effect without the rule being explicitly encoded.

Beyond TD-learning

Modern RL has moved well beyond pure TD. Three directions worth knowing:

  • Policy-gradient methods (REINFORCE, A2C, PPO) directly optimise the policy without an explicit value function. AlphaGo’s policy network is in this family.
  • Actor-critic combines a policy network (actor) with a value network (critic), updating both together. This is the foundation for most modern game-playing systems.
  • MCTS-augmented learning (AlphaZero) interleaves Monte-Carlo Tree Search with neural network evaluation, training the network to imitate MCTS-improved policies. This produced superhuman play in chess and Go.

For backgammon, MCTS is less critical because of the dice — you’d have to sample over dice rolls at every node, which inflates the tree exponentially. So mainline backgammon engines stick close to TD with ply-limited search.

What’s next in this series

This was the foundational primer. The next articles cover the harder questions:

If you want concrete code rather than theory first, the API quickstart and the analysis output reference cover the developer-facing surface that this RL machinery powers.