Swift Transformers matches Hugging Face tokenizers byte-for-byte
The release handles difficult Unicode cases such as Hangul composition, Japanese dakuten, and emoji without drifting from the reference tokenizer behavior
We just released swift-transformers 1.3.3 with byte-per-byte conformance with the referance tokenizers library for complicated multilingual Unicode cases like Hangul combined characters, Japanese voiced-kana dakuten, or emojis such as (pre