# Chess commentary encoder

> A 230M-parameter encoder that reads chess commentary, finds every move, square and claim, and checks each claim against the board. Runs in your browser.

A fine-tuned LFM2.5-Encoder-230M reads chess commentary and tags every move, piece, square and claim; chess.js then checks each claim against the board. I built it to reward true reasoning during RL ([the story](https://www.manoletre.com/chess.md), and [how it works](#how-it-works) below). It runs in your browser: the first load downloads the model (~145 MB) from [Hugging Face](https://huggingface.co/manoletre/chess-move-tagger). Code and data are on [GitHub](https://github.com/manoletre/chess-moves-from-text). Pick an example or write your own.

## how it works

The LLMs I trained to play chess wrote a short reflection before every move, and I wanted to reward reflections that were true. To check one against the board I first need to know what each part of it says: which words are moves, which are pieces and squares, and which are claims like “gives check” or “wins the queen”. That's token classification, which a small encoder does well and fast.

LFM2.5-Encoder-230M turns every token into a vector of 1,024 numbers. On top of it I trained a single linear layer with 36 outputs, split into five heads. Each head is its own classifier over the same tokens, with its own softmax and cross-entropy loss, and the five losses are added up.

*Figure: One sentence, one forward pass: all five heads label every word at the same time. Then each head's labels join into spans, and the spans of the five heads line up into four tagged spans.*

'The rook can swing to g8 and deliver check, but Black's king simply steps away.' mention: B-MOVE I-MOVE ×4 on 'rook can swing to g8', B-PIECE I-PIECE on 'Black's king'. line: new on the move. stance: rejected on the move. frame: after on 'deliver check', 'Black's king', 'simply steps away'. claim: CHECK on 'deliver check', FLIGHT on 'simply steps away'. Together: a rejected move, a check claim, a piece and a king-flight claim, all about the board after the move.

| head | labels | outputs |
| --- | --- | --- |
| mention | move, piece, square, file, rank, diagonal, region | 15 |
| line | new, next, alternative | 3 |
| stance | chosen, rejected, considered | 3 |
| frame | now, after | 2 |
| claim | check, mate, wins, attacks, flight, eval | 13 |

The mention and claim heads have a begin and an inside label per type plus one for “nothing”, so they mark where a span starts and where it continues, and spans snap to whole words. The other three heads label the tokens inside a span, and the span takes the label its tokens agree on most (their log-probabilities averaged). The line head lets me replay what the text says: a new move starts a line, the next ones continue it, and an alternative replaces its last move.

Then rules turn tags into facts. Each claim is linked to the nearest move or piece before it in its clause, and checked on the board as it is now or after that move, depending on its frame: python-chess during training, a chess.js port of the same rules here in the browser (both give the same verdict on all 34,621 facts I compared). A fact is true, false or unverifiable. Evaluations like “the attack fizzles” can only be checked when the solution is known. The checker also reports the chosen move, moves the text rejected and then played, and false refutations: reasons against a move that aren't true.

*Figure: The four spans become three facts on the board. The text rejected Rg8 because the king would step away, but after Rg8 it can't: a false refutation.*

'rook can swing to g8' resolves to Rg8 (true). 'deliver check' after Rg8: true. 'simply steps away' after Rg8: false, the king has no legal move. So the text rejected the solution, Rg8#, for a false reason.

Here is the whole reflection that sentence comes from, one head at a time.

*Figure: The whole reflection through each head, then the checks. Click a head to see its labels. One verdict is the checker's mistake: it linked “aimed at” to f1 as well as g8.*

A real reflection: 'White's rook on g2 is aimed at the g8 square ... So the move is Rg8#.' mention: pieces, squares and 3 moves. line: all 3 moves NEW. stance: the first move REJECTED, the other two CHOSEN. frame: claims about now or after the move. claim: ATTACKS, CHECK, FLIGHT, EVAL, ATTACKS, MATE. verify: 'simply steps away' and 'the attack fizzles' are false reasons against Rg8, the move it then plays.

## data and training

704 training texts are real reflections from the LLMs I trained to play chess, from different points of RL, so the encoder sees how their wording drifted. Claude labelled them following a labelling guide, and they are split by puzzle, so no test puzzle shows up in training. The synthetic ones are LLM-written reflections, paraphrases, reflections with planted false claims, and templates for refutations. The commentary comes from an earlier version that only tagged moves, pieces and squares, so it trains the mention and line heads only.

*Figure: The training set. Validation has 137 texts and test 641, both with real reflections and commentary.*

Train: 3,701 texts: 704 real reflections, 1,497 synthetic reflections, 1,500 chess commentary. Val 137, test 641.

I trained for 8 epochs (AdamW, learning rate 5e-5, batch 16) on my laptop's GPU, about 1.5 hours, and kept the epoch that did best on 62 held-out real reflections (epoch 5).

| test set | mentions F1 | claims F1 |
| --- | --- | --- |
| real reflections (194) | 0.987 | 0.939 |
| synthetic (153) | 0.980 | 0.869 |
| commentary (294) | 0.987 | – |

A span counts only with the exact start, end and type. The 4-bit model in this page scores 0.986 / 0.939 on real reflections.

What matters for the reward is the verdicts: from the encoder's tags, the checker reaches the same verdict as from the hand labels for 95.5% of 5,732 facts, and finds the same chosen move in 98.6% of texts. It is weaker on wording it has never seen: on a probe of new phrasings it finds 97% of moves but only 73% of mate claims. That gap matters during RL, where the model keeps finding new ways to say things.

The model is on [Hugging Face](https://huggingface.co/manoletre/chess-move-tagger), and the code, the labelling guide and the data are on [GitHub](https://github.com/manoletre/chess-moves-from-text).