Teaching chess to small language models

tl;dr

I've been teaching small models to play chess. I started with an LLM that reasons before every move (SFT, then GRPO with a curriculum and a process reward), hit a ceiling on my puzzle benchmark[1], and switched to training Liquid's d1 decision models, where I needed less data and compute to get good results.

Puzzles: share of the 1,000 puzzles in my benchmark[1] solved to the end. Games: score over 40 games per opponent from balanced positions after move 5; sf-N is Stockfish limited to N Elo. The search keeps the model's top 3 moves at each of 4 plies and scores the leaves with its own value head.

every model I finetuned, on the puzzles benchmark

I built the benchmark from the Lichess puzzle database[1]: 1,000 puzzles, 20 for each of 50 themes (mating patterns, tactics, endgames, mates in 3 to 5), spread evenly over five difficulty bands from a 399 to a 2792 rating. None of them comes from a game I trained on, so every model faces positions it has never seen. It's on Hugging Face[2].

LFM2.5-1.2B, prompted       1.5% █
+ SFT                       6.7% ███
+ GRPO                     13.0% █████
+ curriculum GRPO          15.4% ██████
d1-3B, LoRA SFT            16.2% ██████
d1-omni-600M, DAgger       28.2% ███████████
d1-omni-600M, fresh SFT    40.9% ████████████████
  + 3-ply search           60.2% ████████████████████████

part 1: an LLM that reasons before it moves

The first model I trained was LFM2.5-1.2B-Instruct, playing through tools: it asks for the board and the legal moves, writes a short reflection, then makes a move.

SFT on 24k full puzzle conversations taught it that protocol (0% → 100% valid tool calls) and took puzzles from 1.5% to 6.7%.

Then GRPO: +1 for the right move, 0 for a wrong one, −0.5 for an illegal one. Puzzles doubled to 13.0% and illegal moves almost disappeared. After ~300 steps it plateaued, and the model stopped exploring: 8 samples of the same position gave 1.7 different moves.

I thought the model was getting to hard positions too soon, so I tried a curriculum: four stages, from easy to hard. It reached 15.4%, my best number with an LLM. But when I read the reflections, they had collapsed into one template: 91% claimed a mate, in puzzles where 6% had one.

Since it had stopped exploring, I raised the temperature. The reflections turned into garbage (at the temperature that scored best, 87% were garbled text) and a move still came out at the end. That's when I understood the reflections had never meant much: the model picked a move, and the reflection didn't match the choice.

So I tried rewarding reflections that are true. I trained a tagger[1], LFM2.5-Encoder-230M as a token classifier, that finds every move, square and claim in a reflection, and checked each claim with python-chess. False facts fell from 53% to 24% and self-contradictions from 39% to 5%, but puzzles stayed at 13.4%.

And RL kept finding ways around the reward. I closed one hole and it found the next: the last one dismissed every alternative with lines like “lets White consolidate”, true of any wrong move. It became a cat-and-mouse game, and I don't have the compute to play it.

part 2: a decision model

In October Liquid released d1, models that score a list of options instead of writing text. A d1 question over the legal moves is a policy; a question over ten win-chance bins is a value head. So I could train on Stockfish's full ranking of moves for millions of positions.

First, the dumb baseline. d1-3B solved 36.5% of puzzles out of the box, which looked amazing until a rule that only reads the + and # marks in the move names scored 36%. With plain UCI moves it was at chance.

SFT with LoRA on 120k Stockfish-labelled positions (KL to the engine's move distribution) got d1-3B to 16.2%. RL from games on top stayed flat for 28 iterations.

DAgger worked: play games, have Stockfish label the positions the model actually reaches, train on them, repeat. Four rounds cut the win chance lost per move by 30% (4.63 → 3.26 points).

d1-omni-600M trains about 5× faster, so a night fits 11 DAgger rounds: 2.86 points lost per move and 28.2% of puzzles. A small search on top more than tripled its score against Stockfish 1320 (21% → 69%).

Then fresh data beat more DAgger. The last run trained 9 rounds on ~1.6M positions it had never seen, from Lichess's evaluation database plus new games Stockfish labelled on the VM's CPUs while the GPU trained. Loss per move fell from 2.86 to 2.23, puzzles from 28.2% to 40.9% without search, and it was still improving when time ran out. Play against it[1].

what I learned

• Read the samples, not only the metrics. A temperature that looked best wrote garbled text 87% of the time.
• Always compare with the dumbest baseline you can think of.
• RL optimizes the scorer, not what you meant. Never reward a fact that is trivially true.
• Fresh data beats clever rewards and extra epochs at this scale.
• Search on top of a decent policy is cheap strength.

I made narrated explainers of two methods I used, with my explainer-video[1] skill: DAgger[2] and Self-Evolving Curriculum[3].