b1g-fragment02 is only partially annotated in VUAMC — 9,287 unannotated all-POS training tokens carry gold label `0`

## Summary

While auditing the VUA-18 / VUA-20 training data I found that one of the 117 texts, **`b1g-fragment02`**, is annotated for metaphor only in part in the source corpus, but is present in full in the distributed `VUAMC.xml`. Because TEI encoding marks metaphor by the *presence* of a `` element, absence of annotation and a negative judgement look identical to a converter. As a result, `vua_xml_parser.py` assigned gold label `0` to roughly 9,287 all-POS training tokens that were never annotated by anyone.

**This is a format/conversion issue, not an annotation error, and not a mistake in the shared task design.** The corpus creators documented the coded extent correctly; the information simply is not machine-readable from the XML. I am reporting it here because the affected artefacts are the ones distributed from this repository, and because they are still widely used.

**The test partition is unaffected** — `b1g-fragment02` falls entirely inside the training split (0 tokens in `all_pos_tokens_test.csv` and `verb_tokens_test.csv`). Published test scores remain valid; the effect is on what systems could learn, not on how they were measured.

## Affected artefacts

| Artefact | Release | What is affected |
|---|---|---|
| `all_pos_tokens.csv` (72,611 lines) | `naacl_flp_train_gold_labels.zip`, v1.0 | 10,925 tokens from `b1g-fragment02`; **9,287 of them (85.0%) are unannotated but labelled `0`** |
| `verb_tokens.csv` (17,240 lines) | `naacl_flp_train_gold_labels.zip`, v1.0 | 1,717 tokens from `b1g-fragment02`; **1,459 unannotated but labelled `0`** |
| `vuamc_corpus_train.csv` | generated by `vua_xml_parser.py`, `naacl_flp_starter_kit.zip`, v1.0 | the same sentences appear without `M_` prefixes |

The annotated part of `b1g-fragment02` is confined to two contiguous sentence bands, **sentences 738–765 and 1485–1584**. Everything outside those two bands is unannotated.

## Evidence

Each of the following is independent of the others, and none of them uses a model or any training:

1. **The corpus creators' own documentation.** Steen et al. (2010), *A Method for Linguistic Metaphor Identification*, Appendix A, lists **3,006 lexical units** as coded for `b1g`, whereas the distributed section runs to about 20,170 words. Herrmann (2013), Appendix A1, gives the same figure independently.

2. **The distribution of the markup in the XML.** Of the 637 `mrw` segments in `b1g-fragment02`, **634 fall inside the two bands and 3 outside**. The boundary is sharp — annotation does not thin out gradually, it stops.

3. **A second, independent annotation layer.** The `corresp` attributes (a different annotator task from metaphor marking) draw the same boundary: 13 of 16 pairs lie inside the bands.

4. **The published register statistics.** Counting the unannotated part as annotated makes the academic register about **17,238 lexical units larger** than the published corpus description.

5. **The published metaphor rate.** Excluding the unannotated part reproduces the published academic figure (**18.46% against a published 18.5%**); including it does not.

6. **A corpus-wide audit.** All 117 documents were tested with the same procedure for a "two bands plus a sharp boundary" pattern. **Exactly one document shows it: `b1g-fragment02`.** This is a single, localised defect, not a diffuse property of the corpus.

7. **A model-free prior test.** A leave-one-document-out lemma prior, computed only from the gold labels themselves, predicts about **1,634.9 metaphor positives** in the unannotated region. The observed number is **3** (`z = −53.66`). In the Verbs band the unannotated region contains **0** positives out of 1,459 target tokens.

8. **The shared task's own published anomaly.** Table 2 of Leong et al. (2018) reports an academic-register metaphor rate of **14.41% in training against 24.14% in test** — a roughly ten-point gap that has, to my knowledge, never been explained. Removing the unannotated region accounts for it.

## Why the conversion produces this

`vua_xml_parser.py` walks the TEI tree and emits `M_` for every token inside a ``. It has no way to ask whether a given section was in scope for annotation at all, because the XML does not record annotation scope — only annotation results. For a fully annotated document the two are equivalent. For a partially annotated one they are not, and the difference is silently converted into supervision.

## Measured impact

Fine-tuning `bert-base-uncased` as a token classifier on the official split, three seeds per condition, identical recipe throughout:

| Condition | All POS F1 | Verbs F1 |
|---|---|---|
| BASE — as distributed | 0.7070 | 0.7368 |
| TRIM — unannotated region given no supervision | **0.7200** | **0.7575** |
| RAND — size-matched control: the same number of *valid* tokens removed | 0.7040 | 0.7345 |

- All POS: **TRIM − BASE = +0.0131**, 95% CI [+0.0082; +0.0194]; **TRIM − RAND = +0.0161** [+0.0115; +0.0210]; RAND − BASE = −0.0030, CI includes zero.
- Verbs: **TRIM − BASE = +0.0207**, 95% CI [+0.0112; +0.0307]; **TRIM − RAND = +0.0230** [+0.0133; +0.0344]; RAND − BASE = −0.0023, CI includes zero.
- Document-level cross-validation over all 117 texts, evaluated on the token set common to both corpora: **ΔF1 = +0.0081** [+0.0031; +0.0140], with a built-in null control (the fold containing `b1g-fragment02`, where both corpora train on identical data) measuring **+0.0001**.
- Confidence intervals come from a document-level percentile bootstrap (4,000 resamples), so they reflect document sampling rather than token-level pseudo-precision.

For scale: on the corrected data an unmodified `bert-base` reaches the organisers' official BERT baseline (0.718; Leong et al., 2020) with no architectural change. The correction is not decisive for the ranking — I have not re-run any submitted system — but it is the same order of magnitude as differences usually reported as modelling progress.

## What I am not claiming

- Not that the corpus is wrong. The VUAMC annotation is what it says it is; the coded extent is documented.
- Not that the published rankings are invalid. Establishing that would require re-running every system on corrected data, which I have not done.
- Not that the test scores are affected. They are not.

## Possible remedies

Roughly in order of effort, any of which would help:

1. **A note in `VUA-shared-task/README.md`** recording that `b1g-fragment02` is only partially annotated in the source corpus, and that its unannotated tokens carry label `0` in the v1.0 training gold files.
2. **A supplementary exclusion list** — the token IDs of the unannotated region — distributed alongside the existing gold files, so that users can opt into the corrected supervision without breaking reproduction of earlier results.
3. **A revised gold file** in which the affected tokens are marked as unknown rather than negative (for example, a third label value, or simply omitted from the token list).

I would suggest against silently changing `all_pos_tokens.csv` and `verb_tokens.csv` in place: a great deal of published work trained on those exact files, and being able to reproduce the old numbers has value. An additive exclusion list keeps both paths open.

More generally: a conversion step of this kind needs to know the boundaries of the annotated range, and where it does not know them it should emit *unknown*, not zero.

## Reproduction

Everything below is deterministic and needs no GPU:

- the exclusion list (token IDs of the unannotated region, for both bands),
- the two coded sentence-band boundaries,
- the corpus-wide audit script that finds the pattern in 1 of 117 documents,
- the lemma-prior test,
- the register-statistics recomputation,
- the full correction-experiment recipe, per condition and per seed, with SHA-256 manifests of inputs, code and outputs.

Deposited at: **https://doi.org/10.5281/zenodo.22216961**

Primary sources used: `VUAMC.xml` from the Oxford Text Archive (record 2541) and the v1.0 gold files from this repository.

## Related

A manuscript describing the audit is in preparation. I am reporting the finding here first, ahead of submission, so that it can be discussed by the people who maintain the resource before it appears in print. I am happy to prepare a pull request for whichever remedy you consider appropriate, or to supply the exclusion list in whatever format is most useful.

I have sent a parallel note to the VU Amsterdam Metaphor Lab, since the question of the coded extent is ultimately theirs to confirm.

Thank you for maintaining this dataset — the fact that the conversion pipeline is public and reproducible is exactly what made this finding possible.

— Gergely Tóth, toth.gergely@kre.hu