Language data for multilingual AI
Explore the open datasets supporting language model development, training, and evaluation across languages beyond the mainstream.
- OPEN DATA
- MULTILINGUAL AI
- RESEARCH
The open data foundations behind our models
Sunbird SALT
Parallel · Hugging FaceA multilingual parallel corpus covering locally relevant topics across Ugandan languages and Swahili, supporting translation model development.
View DatasetMAFAND-MT (en-swa)
Parallel · GitHubHuman-translated news-domain parallel data supporting English–Swahili machine translation.
View DatasetEnglish–Swahili Sentence Pairs
Parallel · Hugging FaceGeneral-domain parallel sentences supporting English-to-Swahili translation training.
View DatasetInkuba-Swahili-MMT
Parallel · Hugging FaceMultilingual parallel data prepared for English-to-Swahili translation fine-tuning.
View DatasetAya Dataset (Swahili)
Instruction · Hugging FaceHuman-annotated Swahili instruction data supporting multilingual instruction-following.
View DatasetAlpaca-Cleaned (Swahili)
Instruction · Hugging FaceSwahili instruction-tuning data adapted from the cleaned Alpaca dataset.
View DatasetKenSwQuAD
Question Answering · Hugging FaceSwahili question-answering data drawn from Kenyan news content.
View DatasetJamiiForums-2B
Monolingual · Hugging FaceA large Swahili text corpus supporting continued pretraining and language adaptation.
View DatasetInkuba-Mono (Swahili)
Monolingual · Hugging FaceMonolingual Swahili text supporting continued pretraining and language modelling.
View DatasetFLORES+
Evaluation · Hugging FaceA multilingual evaluation benchmark for measuring translation quality using standardised metrics.
View DatasetLuganda LLM Data
Parallel · KaggleA prepared English–Luganda parallel corpus with training, validation, and test splits for translation model development.
View Dataset
