Last month, I participated in the Global South Hackathon by Apart Research. With a small team from across South East Asia, we looked at the differential implications of LLM usage across different cultures, specifically, we looked at token fertility in different Asian languages. It turns out having a majority of training datasets in the English language produces unintentional side effects: the models learn less efficiently by producing more tokens per semantic unit (word or character combination). This high fertility means more input and output token usage per prompt (so low fertility = good!).
We tested 16 South East Asian languages across three tokenizer/model combinations (o200k with GPT-4o, cl100k with GPT4/3.5, llama3 with Llama-3.1/SEA-LION). We found that while the selected languages universally performed worse than English, results varied across the three tokenizers.
Key findings
- All model and tokenizer combinations had approximately 1.5x the fertility of English in languages that used Latin script (Vietnamese, Malay, Filipino, Indonesian). These were the best non-English results we tested.
- GPT 4/3.5 and Llama-3.1/SEA-LION performed similarly across the board, with results ranging from 2x to 11x fertility across the languages tested.
- GPT-4o performed significantly better across the board than the other two (with the exception of Laotian), representing a significant advance in multilingual support that can be measured in more than just quality of output.
To represent the direct impact of this tokenizer “tax” we explored the increased rate of context window exhaustion. As you can see, it happens an order of magnitude sooner.
As cloud hosted LLMs charge on a per token usage basis (or otherwise have usage limits measured in tokens), the cost differences here (and potential savings by using an appropriate tokenizer) are enormous. A monthly workload which might cost $1000USD on one model/tokenizer might cost $11,000USD on a different combination!
Retrospective
We had two priorities as a team in choosing this topic:
- Relevance and use to the South East Asian region
- Underaddressed in the research literature with room for tangible impact
This topic represented a good balance of both, and allowed us to explore deeply on a specific aspect of the LLM training pipeline. We also made heavy use of LLMs to expand and speed up aspects of the project, allowing it to be higher quality and more comprehensive than otherwise. The cons of that approach meant there was nontrivial amount of human review for the final PDF paper to address various formatting and structural problems! In future I would experiment with the rate of LLM automation in different areas, but as a test case for automated research this was a highly productive effort.
Full results
Read the full PDF we submitted here https://drive.google.com/file/d/1klbYy4gXobJ6y3D19TT74qQZClX11Ly8/view?usp=drive_link
With thanks to: Leo Dang, Antoine Pedretti, Miles Whiticker, Vinh Van and the Apart Research hackathon organisers
