Research / Technical Report

Fine-tuning Gemma 3 1B for Luganda: from 0.06 to 13.85 BLEU

QLoRA on a single T4 GPU turns a model with essentially no working Luganda into a functional bidirectional translator, using data derived almost entirely from one source.

  • Luganda
  • QLoRA
  • Low-resource NLP
  • 2 MIN READ

·CommonMind Research

English↔Luganda, held-out test split (100 examples)

BLEU
0.06 → 13.85
chrF++
8.84 → 46.59

The problem

Gemma 3 1B Instruct, like most general-purpose small LLMs, has almost no working knowledge of Luganda out of the box: 0.06 BLEU and 8.84 chrF++ on our English→Luganda eval, indistinguishable from random output. Luganda has roughly 5+ million speakers in Uganda, but the training data mainstream LLMs see for it is vanishingly small compared to English, French, or Swahili.

The question was whether a single consumer-GPU fine-tuning run could close meaningful ground here, given that the largest clean parallel corpus we could find was a fraction of the size typically used for high-resource language pairs.

Data

We built 94,542 training examples from roughly 24,000 clean English–Luganda parallel sentence pairs, drawn primarily from the Sunbird SALT dataset, a professionally translated, multi-way parallel corpus covering agriculture, health, society, and other locally relevant topics. Each pair was expanded into three task types: English→Luganda translation, Luganda→English translation, and open Luganda conversation, then split 90/5/5 into train/validation/test.

Method

QLoRA: a 4-bit NF4-quantized base model with LoRA adapters (rank 16, alpha 32, dropout 0.05) on the attention and MLP projections, trained for a single epoch with an effective batch size of 16, cosine learning rate schedule at 2e-4, and bf16 precision on a single Kaggle T4. The whole run (5,500+ steps) took about 8 hours and produced a 13.05M-parameter adapter, roughly 52 MB on disk.

Training and validation loss tracked each other closely throughout (2.07/2.01 at step 500 down to 1.24/1.26 by step 5000), with no sign of overfitting by the time training stopped.

What we'd try next

The model was trained for a single epoch on data covering agriculture, health, and society topics; both loss curves were still trending down when training stopped. Further epochs and additional sources (JW300, the Kimera corpus) would likely push scores higher, and Luganda→English quality specifically has room to close the gap with the reverse direction, since the base model's English fluency doesn't carry over as directly.