Several defects in `examples/prompts.py` that reach the generated prompts

When setting up the repo to work on something else I ran `examples/create_examples.py` over the extracted `benchmark_data`. While
poking at the output I noticed some of the generated prompts looked off, so I went through `prompts.py` language by language, comparing each template to each other. At this point I've found issue 1. As the next step I requested Claude Opus5 to make a run and concentrate on the findings and flag anything else it can find.

Most of the differences turned out to be intentional, but a handful look like genuine mistakes, and all of them show up in `prepared_data` - so they're in the prompts models actually get. I've listed them roughly by how much I think they matter, with a way to reproduce each one.

I'd be glad to fix any or all of these - see the note at the end.

---

## 1. Italian and Spanish label every few-shot example with the *query's* `output_type`

This is the one I'd flag first. Lines 188 and 312 use the outer `output_type`
inside the loop instead of the loop variable's:

```jinja
{% for x in few_shot -%}
Pregunta: {{ x["question"] }} La respuesta debe ser del siguiente tipo: {{ output_type }}.
Respuesta: {{ x["answer"] }}
{% endfor -%}
```

`x["question"]` and `x["answer"]` change per demonstration, but `output_type` resolves to whatever the *query* asked for, on every iteration. Italian and Spanish are the only two of the 31 languages that do this - the other 29 use `x["output_type"]`.

So every demonstration ends up labelled with a type that contradicts its own answer. The easiest way to see it is that the same demonstration gets a different label depending on which question follows it. Lines 1 and 4 of `prepared_data/pretrained/spanish/dev.jsonl` share an identical five-shot block, but it renders differently in each:

```
line 1: Pregunta: ¿Cómo se llamaba la mascota de la Copa Mundial de Fútbol de 1982?
La respuesta debe ser del siguiente tipo: palabra(s).
Respuesta: Cobi

line 4: Pregunta: ¿Cómo se llamaba la mascota de la Copa Mundial de Fútbol de 1982?
La respuesta debe ser del siguiente tipo: un número.
Respuesta: Cobi
```

On that same line the date `1 de marzo de 1979` is also labelled `un número`.

The fix is one token in each template:

```diff
-Pregunta: {{ x["question"] }} La respuesta debe ser del siguiente tipo: {{ output_type }}.
+Pregunta: {{ x["question"] }} La respuesta debe ser del siguiente tipo: {{ x["output_type"] }}.
```

This is the only item here I'd expect to actually move scores.

---

## 2. Some stray Latin characters got into five templates

Each of these lands in all 250 generated prompts for its language. What convinced me they're accidental rather than deliberate is that in the Arabic and Traditional Mandarin cases the demonstration block is clean and only the final query has the extra character.

| file:line | language | current | expected |
| --- | --- | --- | --- |
| `prompts.py:59` | cantonese | `{% for x in few_shot -%}c` | `{% for x in few_shot -%}` |
| `prompts.py:53` | arabic | `في شكلc` | `في شكل` (as in the demo, line 49) |
| `prompts.py:364` | traditional_mandarin | `只產生一個d` | `只產生一個` (as in the demo, line 359) |
| `prompts.py:60,64` | bengali | `একটিc{{ output_type }}দাও` | `একটি {{ output_type }} দাও` |
| `prompts.py:642` | bengali (chat) | `{{ output_type }}cদাও` | `{{ output_type }} দাও` |

The Cantonese one is the most noticeable. Because `-%}` strips the trailing newline, that `c` gets emitted on every loop iteration, so a five-shot Cantonese prompt opens with five stray `c` lines:

```
c
問題：王菲係邊一年出道？只係產生一個數字就得。同埋要用廣東話口語回答。
答案：1989

c
問題：...
```

I'd guess the Bengali spacing (`একটিc{{ ... }}দাও`, with no spaces either side of the variable) wants a look from someone who reads Bengali, since I can't tell whether the missing spaces are intentional.

---

## 3. Cantonese few-shot mixes traditional and simplified characters

Line 70 uses traditional `問題：` in the demonstration block, but line 75 uses simplified `问题：` in the query.

---

## 4. `FA_TOKENS["khmer"]` doesn't match its own template

```python
"khmer": ("សំណួរ៖c", "ចម្លើយ៖"), # prompts.py:25, and eval.py:41
```

The Khmer templates use the clean `សំណួរ៖` (lines 207 and 724), so that declared question token never actually appears in a generated prompt. Nothing in the repo reads these values today - `create_examples.py` only iterates `.keys()` - but the README suggests assembling prompts yourself, and these are the obvious stop sequences.

Worth mentioning that `FA_TOKENS` is duplicated verbatim in `eval.py:26` and `prompts.py:10`, so this typo needs fixing in two places. Importing one from the other would stop the copies drifting apart later.

---

## 5. Things I'm less sure about - probably want a native speaker

These might well be deliberate; I couldn't tell from the code.

Japanese and Korean sentence-final punctuation differs between demo and query, in opposite directions. Japanese demos end with an ASCII `.` (line 199) while the query uses `。` (line 204). Korean is the reverse: demo uses `.` (line 222), query uses `。` (line 227), even though Korean normally takes the ASCII period. The Korean chat template at line 732 uses `.`, which makes line 227 look like the outlier.

German uses different verbs in the two positions. Line 155 has `Gib deine Antwort nur als {{ x["output_type"] }}.` and line 160 has `Schreibe deine Antwort als {{ output_type }}.` - different verb, and the query drops `nur` ("only"), which is the word carrying the brevity instruction.

I should say that a demo/query wording difference is *expected* in several languages and isn't a bug on its own: `fewshot_examples` stores bare types (`number`) while `dev.jsonl` stores articled ones (`a number`), so English correctly writes `Produce only a {{ x["output_type"] }}` in the demo and `Produce only {{ output_type }}` in the query. German goes beyond that, which is why it caught my eye.

One more small thing that's in the data archive rather than in `prompts.py`: the English few-shot types are `number, name, Number, date, year`, with the third capitalised, so one demo reads `Produce only a Number.`

---

## Things I checked that turned out fine

Noting these so nobody has to repeat the work:

The hardcoded `fewshot_examples` in `prompts.py` match `benchmark_data/{lang}/knowledge_fewshot.jsonl` exactly - same questions, answers and types - in every language I sampled. They're duplicated rather than read from the archive, which could drift if either side is edited, but right now they agree.

There's no few-shot contamination: none of the five demonstration questions appears in the corresponding `dev.jsonl`.

Structurally the file is in good shape. All four dicts cover the same 31 languages, every language has exactly five few-shot examples with `question`/`output_type`/`answer` keys, and all 62 templates render without error.

Also, `dev.jsonl` is ordered in contiguous runs by `output_type` - Spanish goes 3 `palabra(s)`, then 33 `un número`, then 2 `otro`, then 66 `un nombre`, and so on. Not a problem in itself, but it does mean taking the first N rows gives you a very skewed answer-type distribution, which is worth knowing if anyone subsamples.

---

## Reproducing

```bash
python3 -m venv .venv && .venv/bin/pip install -r requirements.txt jinja2
openssl enc -aes-256-cbc -pbkdf2 -iter 100000 -d \
-in benchmark_data.tar.xz.enc -pass pass:multiloko | tar -xJ
cd examples && ../.venv/bin/python create_examples.py
```

For item 1:

```bash
head -2 ../prepared_data/pretrained/spanish/dev.jsonl
```

and for the Cantonese stray character in item 2:

```bash
python3 -c "import json; print(json.loads(open('../prepared_data/pretrained/cantonese/dev.jsonl').readline())['text'][:60])"
```

---

## Happy to do the work

I'd be glad to fix any or all of these, whichever subset you're comfortable with. Items 1 and 2 are mechanical, so I can send those as small separate PRs whenever you like, and item 4 is nearly as simple.

For item 3 and anything in item 5 I'd rather not guess: if a native speaker on your side can confirm what the Cantonese, Japanese, Korean, German and Bengali templates should say, I'm happy to make those changes too.

The one thing I wanted to flag before sending patches is that any of this changes the generated prompts, so scores produced afterwards wouldn't be directly comparable with the published leaderboard or the paper. That felt like your call which is why I've opened an issue instead. Just let me know how you'd like to handle it.

Start with examples/prompts.py and the duplicated FA_TOKENS definitions in examples/eval.py, focusing on the listed lines and template differences. Run examples/create_examples.py using the documented setup, then inspect the Spanish and Cantonese generated files with the supplied commands. Done means the confirmed prompt defects are corrected, generated output is clean, and the uncertain language-specific wording has been resolved with maintainer or native-speaker guidance.

Written by the indexing model from the issue text.