BrahmicTokenizer-131K: An Indic-Capable Drop-In Replacement for o200k_base 文章

ArXiv CS.CL2026-05-29NEWSen作者: Rohan Shravan
查看原文 →
关系图谱

BrahmicTokenizer-131K: An Indic-Capable Drop-In Replacement for o200k_base · 相关技术

相关技术

linear-programming allocationBrahmic compressionbyte-level BPEbyte-level BPE tokenizer