Benchmarks

Measure what improves

Open reproducible benchmarks for evaluating multilingual AI performance across languages, civic discourse tasks, and community contexts.

  • MULTILINGUAL
  • OPEN EVALUATION
  • CIVIC AI

Compare Model Performance

Explore and compare model performance across languages, tasks, and evaluation metrics, with transparent results and references to public sources.

ModelScore · BLEU CommonMind Other
ModelScore · chrF++ CommonMind Other

FLORES-200 devtest · 1,012 sentence pairs

Compare Models

Sample data

Compare selected models across evaluation tasks and metrics. Translation results reflect measured chrF++ scores, while other tasks use illustrative values pending publication of CivicBench Africa results.

  • Luganda-gemma-1B-it
  • Swahili-gemma-1B-it
  • Swahili-nllb-1.3B
CivicBench Africa

Nine dimensions of civic understanding

Civic bench Africa establishes a framework for evaluating how AI models interprete, process, and respond to civic discourse across languages and community contexts.

  1. 01Civic Speech RecognitionTranscription fidelity across community voice reports and public meeting recordings.
    WER · CERIn development
  2. 02Code-SwitchingRobustness across transitions between local languages and English within the same conversation.
    WER on switched spansIn development
  3. 03Civic TerminologyMapping locally expressed civic concepts to shared meanings across languages and contexts.
    Accuracy · top-kIn development
  4. 04TranslationPreservation of meaning between English and local languages on standardised evaluation sets.
    BLEU · chrF++Results
  5. 05Issue ClassificationClassification of community reports across civic domains, including public services, elections, and governance.
    Macro-F1In development
  6. 06Entity RecognitionExtraction of institutions, public services, official roles, and geographic references from civic discourse.
    Span F1In development
  7. 07Geographic InterpretationResolution of local place references, from villages and parishes to administrative boundaries.
    Accuracy by administrative levelIn development
  8. 08Claim and Context AssessmentIdentification of claims requiring additional evidence, verification, or contextual interpretation.
    Calibrated F1In development
  9. 09Report SummarisationFaithful synthesis of community reports and public discussions, distinguishing concerns raised from responses provided.
    Faithfulness · ROUGE-LIn development
Adaptation

One foundation. Different language capabilities

Compare Gemma 3 1B before and after language-specific adaptation for multilingual civic AI.

0.00→0.00Luganda · BLEU · SALT held-out0.00→0.00Swahili · BLEU · FLORES-200
Efficiency

Smaller models. Measurable gains

A 615M-parameter model outperforms Gemma 3 27B on English-to-Swahili translation, demonstrating that model size alone does not determine performance.

0102030400.5B1B2B4B10B27Bparameters (log)BLEUGemma 3 1B (base) · 0.4Gemma 3 4B · 10.9Swahili-gemma-1B (Crane) · 27.6Swahili-gemma-1B-it · 28.9Gemma 3 27B · 29.4Swahili-nllb-600M · 36.0Swahili-nllb-1.3B · 38.7
GPT-5 Mini and Gemini 2.0 Flash not plotted: size undisclosed.
Methodology

How we measure

Evaluation Metrics

BLEU measures n-gram overlap with reference translations, while chrF++ captures character- and word-level similarity, providing additional sensitivity to morphological variation.

Standardised Evaluation

Models are compared using shared evaluation sets. Swahili models are evaluated on FLORES-200, while Luganda models use a held-out test set.

Reproducible Results

Evaluation methods, datasets, and model configurations are documented to support transparent, consistent, and reproducible comparisons.

Playground

Explore the models in action

Test how models translate civic discourse between English, Luganda, and Swahili.

English0/500
Luganda
The translation will appear here