Purpose-built beats general-purpose: NLLB LoRA vs. instruction-tuned LLMs for Swahili

Why try a non-chat architecture

Every other CommonMind model so far has been a fine-tuned instruction-tuned LLM (Gemma 3). NLLB-200 is a different kind of model entirely: an encoder-decoder architecture with a SentencePiece tokenizer, built by Meta specifically for translation across 200 languages, with no chat template and no general-purpose reasoning ability to spare. The question: does specializing the architecture, on top of the training data, buy anything on top of what instruction-tuned LLMs already achieve?

Method

LoRA (rank 16, alpha 32) on facebook/nllb-200-distilled-1.3B and facebook/nllb-200-distilled-600M, trained one direction only, English→Swahili (eng_Latn→swh_Latn), on 145,707 parallel pairs: all of MAFAND-MT (32,414) and Sunbird SALT (23,947), plus Rogendo capped at 60,000 and Inkuba-Swahili-MMT capped at 30,000. 3 epochs, 3e-5 learning rate, effective batch size 32, max sequence length 128, on a single Kaggle T4, about 13,662 steps per model.

One honest wrinkle: raw training loss for both runs plateaued far higher (roughly 10–12) than the causal-LM losses we typically see finishing around 1.2–1.5 on our Gemma fine-tunes. That reflects a different loss landscape: NLLB's ~256K-token vocabulary changes what the raw cross-entropy number means, which is why we evaluate held-out translation quality directly.

How it stacks up

Scored on the same FLORES-200 devtest split (1,012 sentence pairs) used across our Swahili benchmarks, the 1.3B NLLB adapter reached 38.74 BLEU / 63.37 chrF++, and the 600M variant reached 35.97 / 61.26. Both beat Crane AI Labs' Swahili-Gemma-1B (27.6 / 56.8), Gemma 3 27B (29.4 / 60.0), and GPT-5 Mini (31.8 / 62.4) on the same benchmark, and the 1.3B model edges past Gemini 2.0 Flash (35.6 BLEU) too. The full comparison, with every source linked, is on the benchmarks page.

Neither model was trained or evaluated on the reverse direction (Swahili→English); that's untested and not claimed.