Train A BPE Tokenizer
I built train_tokenizer: a CLI that trains and evaluates a byte-level BPE tokenizer. It has 100 rows of sample data and a test file.
What a tokenizer does
A language model does not read text. It reads a list of numbers. A tokenizer turns text into that list, and turns the list back into text.
The simplest tokenizer splits text on spaces, into words, and gives each word a number. This breaks fast. Any word the model has not seen has no number. A model that never saw “photosynthesizing” cannot represent it.