·CommonMind Research
English→Swahili, FLORES-200 devtest
- BLEU
- 0.41 → 28.86
- chrF++
- 14.28 → 54.77
Two stages
We split the work in two. Stage 1 continues pretraining the base model on unlabeled Swahili text (JamiiForums-2B, a large monolingual Swahili forum corpus, and Inkuba-Mono, a Swahili monolingual dataset) with no instruction format at all, just raw next-token prediction. That produces an intermediate checkpoint that reads Swahili more fluently before it ever sees a translation pair.
Stage 2 is where translation ability is actually taught: QLoRA fine-tuning (rank 32, alpha 64, dropout 0.05) on top of the CPT checkpoint, for 3 epochs at a 2e-4 cosine schedule, effective batch size 32, packed to 512-token blocks. The adapter was then merged into full fp16 weights, the form the released model ships in.
Data
159,311 examples after quality filtering: 32,414 from MAFAND-MT (human-translated news, via Masakhane), 58,052 from Rogendo's English-Swahili sentence pairs, 23,947 from Sunbird SALT, 29,991 from Inkuba-Swahili-MMT, plus smaller instruction-following sets (359 from Cohere's Aya dataset and 14,844 from Alpaca-cleaned-Swahili) so the model picks up general Swahili instruction-following alongside translation. That's 25,452 packed training blocks, roughly 13 million training tokens, over 2,388 steps and about 9 hours 45 minutes on a single T4.
One thing that didn't work: we planned to add back-translated monolingual Swahili (via NLLB) to the mix, but the monolingual source returned zero usable lines in this run; that augmentation simply didn't contribute to the released model, and we're noting it here.