NorEval is a multi-task Norwegian language understanding and generation evaluation benchmark.
10.11.2026: 🛠️ We release NorEval v1.3, which fixes prompts, uses the filtered version 2.0 of NRK-Quiz-QA, makes length-normalized accuracy the primary metric of the multiple-choice question-answering tasks and SLIDE, and adds task metadata.10.11.2026: 🧩 We release NorEval v1.2, which evaluates every multiple-choice task in three formulations (cloze, hybrid and multiple choice), adds new metrics and samples the generation tasks.03.11.2026: 📌 We release NorEval v1.1.1, which pins every dataset to a fixed version for reproducible evaluation and fixes the Tatoeba English ↔ Bokmål/Nynorsk tasks.17.10.2026: 🆕 We release NorEval v1.1 with new tasks (SLIDE, NoCoLA, MultiBLiMP, NorOpenBookQA without facts, Bokmål ↔ Nynorsk and Bokmål ↔ Northern Sámi translation) and updated task configs.15.10.2026: 📊 Our interactive dashboard is available here.16.05.2025: 🎉 Our paper is accepted to ACL 2025 Findings.07.05.2025: 🫰 NorEval is now integrated into LM Evaluation Harness.10.04.2025: 📕 Our pre-print is available on arXiv.09.04.2025: 🚀 We release NorEval v1.0, including our annotation guidelines and novel datasets (NorRewrite-Instruct & NorSummarize-Instruct).
NorEval is a multi-task Norwegian language understanding and generation evaluation benchmark that combines existing peer-reviewed datasets with new datasets manually created from scratch (NCB, NorRewrite-Instruct, NorSummarize-Instruct for Norwegian Bokmål, and NorIdiom for Norwegian Bokmål and Nynorsk). NorEval covers diverse task categories: sentiment analysis, Norwegian language knowledge, Norwegian-specific & world knowledge, machine reading comprehension, commonsense reasoning, machine translation, text summarization, instruction following, and truthfulness. Our main evaluation principles are:
- 🌐 Linguistic diversity: support for both of the official written standards of Norwegian: Bokmål and Nynorsk (the minority variant).
- 📊 Task diversity: coverage of various least addressed tasks for Norwegian. In particular, only three out of 24 NorEval datasets are included in existing Norwegian benchmarks to date: NorBench, NLEBench, ScandEval, and SEB.
- 🧠 Data quality: focus on only peer-reviewed human-created datasets to ensure reliable evaluation in the context of the Norwegian language, culture, and values.
- 📏 Prompt sensitivity: evaluation across multiple human-written prompts to account for the prompt sensitivity.
- 👩🏻🔬 Standardized evaluation: integration of NorEval into LM Evaluation Harness for flexible and reproducible evaluation.
Explore the results of benchmarking Norwegian and multilingual models on NorEval in our interactive dashboard. The dashboard lets you compare models on individual tasks or category averages across 0-shot, 1-shot, and 5-shot regimes, choose how the scores are aggregated over prompts and normalized, and filter models by size or by whether they are fully open.
The task names end with the language code: nob (Bokmål), nno (Nynorsk), sme (Northern Sámi), eng (English); a translation task name lists the source language first. Each name runs all prompts of the task and, for multiple-choice tasks, all task formulations.
| Dataset | Tasks | Type |
|---|---|---|
| Linguistic knowledge | ||
| NCB | ncb_nob |
Multiple choice |
| NoCoLA | nocola_nob |
Multiple choice |
| SLIDE | slide_nob_nno_swe_dan |
Multiple choice |
| MultiBLiMP | multiblimp_noreval_sme |
Multiple choice |
| NorIdiom | noridiom_nob, noridiom_nno |
Generation |
| ASK-GEC | ask_gec_nob |
Generation |
| Language understanding | ||
| NoReC Sentence | norec_sentence_nob |
Multiple choice |
| NoReC Document | norec_document_nob |
Multiple choice |
| NorOpenBookQA | noropenbookqa_nob, noropenbookqa_nno |
Multiple choice |
| NorBelebele | norbelebele_nob |
Multiple choice |
| NorQuAD | norquad_nob |
Generation |
| World knowledge & reasoning | ||
| NRK-Quiz-QA | nrk_quiz_qa_nob, nrk_quiz_qa_nno |
Multiple choice |
| NorOpenBookQA (without facts) | noropenbookqa_no_fact_nob, noropenbookqa_no_fact_nno |
Multiple choice |
| NorCommonsenseQA | norcommonsenseqa_nob, norcommonsenseqa_nno |
Multiple choice |
| NorTruthfulQA | nortruthfulqa_mc_nob, nortruthfulqa_mc_nno |
Multiple choice |
| NorTruthfulQA | nortruthfulqa_gen_nob, nortruthfulqa_gen_nno |
Generation |
| Generation & summarization | ||
| NorSumm | norsumm_nob, norsumm_nno |
Generation |
| NorRewrite-Instruct | norrewrite_instruct_nob |
Generation |
| NorSummarize-Instruct | norsummarize_instruct_nob |
Generation |
| Translation | ||
| NorSumm Translation | translation_nob_nno, translation_nno_nob |
Generation |
| Tatoeba | translation_eng_nob, translation_nob_eng, translation_eng_nno, translation_nno_eng |
Generation |
| Sámi Tatoeba | translation_nob_sme, translation_sme_nob |
Generation |
The tasks are plain LM Evaluation Harness configs, meaning no fork of the lm-evaluation-harness is needed. Install the harness, clone this repository, and point the harness to it with --include_path:
pip install lm-eval # or lm-eval[vllm] for the vLLM backend
git clone https://github.com/ltgoslo/noreval.gitExample: zero-shot evaluation on NorQuAD across all five prompts:
lm_eval \
--model hf \
--model_args pretrained=norallm/normistral-7b-warm \
--include_path noreval \
--tasks norquad_nob \
--output results/norquad_nob/0-shot/ \
--show_config \
--write_out \
--batch_size auto \
--num_fewshot 0- The task names in the table above run all variants of a task at once. To run a single variant, append the prompt index (e.g.
norquad_nob_p2); for multiple-choice tasks, also add the task formulation,cf,hybridormcf(e.g.norbelebele_nob_mcf_p2, ornorbelebele_nob_mcffor all prompts of one formulation). - Tasks loaded via
--include_pathtake precedence over any same-named NorEval tasks bundled with the harness, so this repository is the source of truth for the task definitions.
guidelines/: the annotation guidelines for the human baselines, prompt writing, and the creation of NorRewrite-Instruct and NorSummarize-Instruct.tools/: example task groups for aggregating results by category and a script for processing and plotting the results.noreval_metadata.json: the category, language(s), primary metric, random baseline and maximum score of each task, with the names of its variants (e.g., for min-max normalization and grouping). It is generated from the task configs withtools/build_metadata.py.
@inproceedings{mikhailov-etal-2025-noreval,
title = "{N}or{E}val: A {N}orwegian Language Understanding and Generation Evaluation Benchmark",
author = "Mikhailov, Vladislav and
Enstad, Tita and
Samuel, David and
Farseth{\r{a}}s, Hans Christian and
Kutuzov, Andrey and
Velldal, Erik and
{\O}vrelid, Lilja",
editor = "Che, Wanxiang and
Nabende, Joyce and
Shutova, Ekaterina and
Pilehvar, Mohammad Taher",
booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
month = jul,
year = "2025",
address = "Vienna, Austria",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2025.findings-acl.181/",
doi = "10.18653/v1/2025.findings-acl.181",
pages = "3495--3541",
ISBN = "979-8-89176-256-5",
}