Seminar: building an Arabic language model through continuous pretraining

The next announcement from the Makarov lab seminar: this time we turn to the Arabic language.

Building an Arabic language model through continuous pretraining
Artem Lipinski (Saint Petersburg State University)

Arabic is one of the world's largest languages by number of speakers, yet modern language models handle it noticeably worse than English or Chinese. The reason is simple: high-quality Arabic data in the public domain is scarce, and existing models were trained predominantly on English texts.

The talk presents work on an Arabic language model that understands and generates meaningful Arabic text while remaining compact enough for practical use.

The talk covers:

• how to collect and prepare a large, high-quality text corpus from scratch;

• why mixing languages in the training data helps rather than hurts;

• the full pipeline for adapting an LLM to a new language, from data collection to SFT — including how to work through each stage efficiently and which mistakes to avoid.

The results are relevant to anyone adapting LLMs to low-resource languages: the approaches apply well beyond Arabic.