chess / commentary encoder
A fine-tuned LFM2.5-Encoder-230M reads chess commentary and tags every move, piece, square and claim; chess.js then checks each claim against the board. I built it to reward true reasoning during RL (the story, and how it works below). It runs in your browser: the first load downloads the model (~145 MB) from Hugging Face. Code and data are on GitHub. Pick an example or write your own.
how it works
The LLMs I trained to play chess wrote a short reflection before every move, and I wanted to reward reflections that were true. To check one against the board I first need to know what each part of it says: which words are moves, which are pieces and squares, and which are claims like “gives check” or “wins the queen”. That's token classification, which a small encoder does well and fast.
LFM2.5-Encoder-230M turns every token into a vector of 1,024 numbers. On top of it I trained a single linear layer with 36 outputs, split into five heads. Each head is its own classifier over the same tokens, with its own softmax and cross-entropy loss, and the five losses are added up.
| head | labels | outputs |
|---|---|---|
| mention | move, piece, square, file, rank, diagonal, region | 15 |
| line | new, next, alternative | 3 |
| stance | chosen, rejected, considered | 3 |
| frame | now, after | 2 |
| claim | check, mate, wins, attacks, flight, eval | 13 |
The mention and claim heads have a begin and an inside label per type plus one for “nothing”, so they mark where a span starts and where it continues, and spans snap to whole words. The other three heads label the tokens inside a span, and the span takes the label its tokens agree on most (their log-probabilities averaged). The line head lets me replay what the text says: a new move starts a line, the next ones continue it, and an alternative replaces its last move.
Then rules turn tags into facts. Each claim is linked to the nearest move or piece before it in its clause, and checked on the board as it is now or after that move, depending on its frame: python-chess during training, a chess.js port of the same rules here in the browser (both give the same verdict on all 34,621 facts I compared). A fact is true, false or unverifiable. Evaluations like “the attack fizzles” can only be checked when the solution is known. The checker also reports the chosen move, moves the text rejected and then played, and false refutations: reasons against a move that aren't true.
The rook can swing to g8 and deliver check, but Black's king simply steps away.
- ✓ “rook can swing to g8” (a move, rejected): White's only rook that can reach g8 is on g2, so it means Rg8
- ✓ “deliver check” (check, after the move): after Rg8 the black king on f8 is in check
- ✗ “simply steps away” (the king escapes, after the move): after Rg8 the king has no legal move: the rook covers e8 and g7, the bishop covers f7 and protects the rook on g8, and e7 holds its own bishop
false refutation: the text rejected Rg8 for a reason that isn't true. Rg8 is mate, the puzzle's solution.
Here is the whole reflection that sentence comes from, one head at a time.
White's rook on g2 is aimed at the g8 square, and the black rook on f1 is too far to interfere. The rook can swing to g8 and deliver check, but Black's king simply steps away and the attack fizzles. The bishop on b3 already controls g8, so the rook can land there and deliver immediate checkmate. So the move is Rg8#.
data and training
704 training texts are real reflections from the LLMs I trained to play chess, from different points of RL, so the encoder sees how their wording drifted. Claude labelled them following a labelling guide, and they are split by puzzle, so no test puzzle shows up in training. The synthetic ones are LLM-written reflections, paraphrases, reflections with planted false claims, and templates for refutations. The commentary comes from an earlier version that only tagged moves, pieces and squares, so it trains the mention and line heads only.
I trained for 8 epochs (AdamW, learning rate 5e-5, batch 16) on my laptop's GPU, about 1.5 hours, and kept the epoch that did best on 62 held-out real reflections (epoch 5).
| test set | mentions F1 | claims F1 |
|---|---|---|
| real reflections (194) | 0.987 | 0.939 |
| synthetic (153) | 0.980 | 0.869 |
| commentary (294) | 0.987 | – |
A span counts only with the exact start, end and type. The 4-bit model in this page scores 0.986 / 0.939 on real reflections.
What matters for the reward is the verdicts: from the encoder's tags, the checker reaches the same verdict as from the hand labels for 95.5% of 5,732 facts, and finds the same chosen move in 98.6% of texts. It is weaker on wording it has never seen: on a probe of new phrasings it finds 97% of moves but only 73% of mate claims. That gap matters during RL, where the model keeps finding new ways to say things.
The model is on Hugging Face[1], and the code, the labelling guide and the data are on GitHub[2].