Production mobile AI apps serving global users face a thorny problem: subword tokenizers trained on Latin-script corpora compress non-Latin text poorly, inflating token counts by 2–4× for Arabic or Chinese. This kills inference speed and blows context windows. The naive fix—shipping separate models per language—doubles binary size and complicates updates. The better solution: hot-swap tokenizers at runtime while keeping a single model in memory.
Why Tokenizer Choice Matters for Non-Latin Scripts
Byte-Pair Encoding (BPE) and SentencePiece tokenizers learn subword units from training corpora. GPT-style models trained predominantly on English create vocabularies optimized for Latin character sequences. When you feed Arabic text like "مرحباً بك في التطبيق" into a GPT-2 tokenizer, it fragments aggressively—often one token per two characters—because Arabic ligatures and diacritics were underrepresented during vocabulary construction.
Measured on a 7B-parameter LLM running llama.cpp on an iPhone 14 Pro, English averages 0.75 tokens per word; Arabic averages 2.1 tokens per word for the same semantic content. At 20 tokens/sec throughput, an Arabic user waits 2.8× longer for identical responses. Context windows shrink proportionally: a 2048-token limit holds ~2700 English words but only ~970 Arabic words.
Vocabulary Overlap and Compression Efficiency
Tokenizers trained on multilingual corpora (mBERT, XLM-R) achieve better balance but still favor high-resource languages. A 50k-token vocabulary might allocate 35k slots to Latin-derived subwords, 8k to CJK, 4k to Arabic/Cyrillic, 3k to other scripts. For a chat app serving Ramallah and London equally, this imbalance directly impacts user experience and server costs (more tokens = more compute).
Architecture: Tokenizer as a Swappable Component
The core insight: tokenizers are stateless encoders/decoders. If your model accepts token IDs as input and emits token IDs as output, you can swap the text↔ID mapping layer without touching model weights. The challenge is managing vocabulary mismatches and memory overhead.
Vocabulary Remapping Table
Start with a base tokenizer (e.g., GPT-2's 50k BPE vocab) embedded in your model. Train auxiliary tokenizers for target scripts—Arabic-optimized BPE with 32k vocab, Hanzi-optimized with 28k vocab. Build a remapping table that translates auxiliary token IDs to base token IDs where subwords overlap, and maps novel subwords to a reserved [UNK] range in the base vocab.
Example: Arabic tokenizer produces token 4521 for "الت". Your remapping table checks if this subword exists in base vocab (it doesn't), so it maps to base token 50001 (first slot in a 2048-token extension range). The model sees 50001; during decoding, you reverse-map 50001 → 4521 → "الت".
This approach requires extending your embedding matrix by 2048–4096 tokens (8–16 MB for FP16 weights). You fine-tune these new embeddings on multilingual data for 2–5k steps while freezing the base model, giving the LLM a chance to learn representations for novel subwords.
Runtime Switching Logic
Detect user locale via system settings or explicit selection. Load the corresponding tokenizer from a memory-mapped file (SentencePiece models are ~500 KB each). The sequence becomes:
- User types Arabic text
- Encode with Arabic tokenizer → auxiliary token IDs
- Remap to base token IDs via lookup table
- Feed to model inference
- Decode base token IDs to auxiliary token IDs
- Decode with Arabic tokenizer → text
Total latency overhead: 0.3–0.8 ms for remapping on a 128-token input (measured with a flat hash map in Swift). Negligible compared to 2–8 second inference times for 7B models on mobile.
Memory and Binary Size Trade-offs
Shipping three tokenizers (Latin, Arabic, CJK) adds ~1.5 MB to your app bundle. The remapping table for 8k novel tokens stored as uint16 pairs is 32 KB. Extended embedding layers add 12 MB (3×2048 tokens × 2 bytes × 2048 hidden dims). Compare this to shipping three full model copies: 3×4 GB = 12 GB. The math is obvious.
At runtime, keep only the active tokenizer in memory. Use mmap for tokenizer files so the OS can page them out under pressure. The remapping table stays resident (32 KB is trivial). Extended embeddings live in the same memory-mapped model file as base weights, so no additional RAM cost beyond the one-time fine-tuning investment.
Practical Constraints
This pattern works when your novel subwords represent