Deze pagina is nog niet volledig in het Nederlands vertaald. Lees ze in het Engels of het Frans: EnglishFrançais
Evaluation
What we measured on real public documents, and where it fails.
Run on 21 October 2026 with the model in the beta: Gemma 4 E4B, 4-bit, unchanged by us. Reference answers were all made by people: official EU summaries and translations, court anonymisation labels, court metadata, human answers to reading questions. One run per item, no cherry-picking.
642 test items · one greedy run, no repeats · 95 % intervals are wide because the sets are small (15 to 60 items per cell) · model file sha256 85a896a0…fab87
Results, base model, 21 October 2026
Higher is better. “Not measured” means we had no reference made by people for that language; it does not mean the feature fails there.
| Task | What was scored | FR | NL | DE | EN | How to read it |
|---|---|---|---|---|---|---|
| Court metadata (dates, courts, case numbers) | Fields exactly right (348 fields) | 0,93 | 0,84 | 1,00 (n = 15) | not measured | Dates and courts: all right. Case numbers are the weak field: Dutch 23 of 37, French appeal numbers 32 of 40. The model sometimes takes a neighbouring number or ignores the expected format. German: 15 items only. |
| Finding names to black out (persons only) | Share of names found / share of findings that were names (614 names) | 0,90 / 0,88 | 0,74 / 0,90 | 0,88 / 0,96 | 0,93 / 0,97 | Dutch is the weak spot: one list of 24 judges was cut off and scored 0. French had 8 false hits on case and company names in one document. E-mail addresses, phone numbers, IBANs and birth dates could not be measured on real text: no labelled source exists. The app also runs a rule layer this test did not include. |
| Translation of EU texts | chrF against the official translation (180 paragraphs; column = target language) | 71,1 | 62,2 | 60,4 | 69,1 | Target language right in 180 of 180. Official EU terminology is where Dutch and German lose points. One reference translation understates quality. |
| Summaries of EU acts (250 words) | Share of the official summary’s figures and references kept (160 summaries) | 0,46 | 0,44 | 0,47 | 0,45 | Summaries keep less than half of the identifiers: use cited answers for figures. Invented numbers: 6 in 160 summaries (3.7 %). Length and language right in 97.5 to 100 %. |
| Cited answers (German, non-legal text) | Correct answers; refusals when the answer is absent (30 questions) | not measured | not measured | 22 of 23 correct; 7 of 7 refusals (read by us) | not measured | One real error: the model named the second-to-last item instead of the last. The automatic scorer counted only 3 of 7 refusals because it missed the phrase “nicht verfügbar”: a gap in our scorer, not a model error. |
| Cited answers (earlier test, Swiss law, 40 questions) | Correct; citation supports the answer | 87.5 % correct (English 100 %, French 75 %); 87.5 % of citations support the answer | About 1 answer in 8 was wrong. Open the cited page every time. | |||
Not measured, so do not assume it works
English extraction; cited answers in French, Dutch and English on real text; deadline extraction; redaction of e-mail addresses, phone numbers, birth dates and IBANs on real documents; speech on real voices; speed on Windows laptops without a graphics chip.
Our own synthetic tests overstated the model
On test data built the same way as our training data, the same model scored higher than on the real documents above: extraction 0.99 against 0.91, translation 72 against 66, summaries 0.60 against 0.46. That is why this page uses real documents.
What the results point to
Fixes identified, not shipped yet: accept answers wrapped in code fences and snap values to the expected list (about a third of the case-number misses are format only), recognise more refusal phrasings, and allow longer output for long lists of names. A fine-tuned model is being prepared; if it ships, its results will be published here next to these, with the same weak-spots column.
Caveats
One sample per item, no variance across runs. The summary and translation sets use the same 40 acts in four languages, so their columns are not independent. Extraction items were limited to decisions whose metadata appears in the text, which favours well-formed headers. EU texts and Wikipedia are almost certainly in the model’s pretraining data, so those tasks may be easier than unseen text. Label conventions (titles, initials) explain most of the gap between strict and relaxed matching in redaction; we report the relaxed figures.
Sources of the reference answers: EUR-Lex-Sum (CC BY 4.0), MAPA (CC BY 4.0), Text Anonymisation Benchmark (MIT), Rechtspraak open data, Cour de cassation decisions (DILA, Licence Ouverte 2.0), Open Legal Data, GermanQuAD (CC BY 4.0).
Installeren, een bestand openen, klaar.
Geen kaart, geen account. De proef duurt 15 dagen en bevat elke functie.
De 15 dagen beginnen wanneer u in de app op Start drukt, niet bij het downloaden.
Kunt u geen software installeren op uw kantoorcomputer? Lees de beveiligingsfiche samen met uw IT.
Bèta · installatiebestanden nog niet ondertekend · Wijzigingslogboek