suBPEriod - Byte Pair Encoding at speed
September 21, 2026
suBPEriod is a C++ Byte Pair Encoding encoder I wrote so I could work on speed without giving up accuracy. Unit tests require the same token ids as tiktoken, including Unicode, and a fuzzer checks that odd inputs do not hang or crash. I kept a change only when a second run and the profiles agreed. On a 100 megabyte English sample it used about half of tiktoken's time, and Chinese stayed near parity.