VOCEvaluator's annots.pkl cache is never invalidated — root cause of the recurring "KeyError: <imagename>" reports (#1397, #874, #1490)

yolox/evaluators/voc_eval.py's voc_eval() builds a ground-truth annotation cache the first time eval runs for an experiment, then trusts it unconditionally on every subsequent eval - there's no check that it still matches the current dataset. Iterating on a custom dataset (adding images, re-labeling, re-tiling) between training runs without manually deleting <data_dir>/annotations_cache/ causes the next eval to crash with KeyError as soon as it hits an image that wasn't present when the cache was written.

This is almost certainly the real, never-diagnosed cause of several long-open issues on this repo - #1397 ("when i train in epoch = 2, find this KeyError, what mistake did i make?"), #874, and #1490 ("what is annots.pkl?"). None of them identify the stale-cache mechanism; affected users are just left to manually delete the cache folder as an undocumented workaround, or get no answer at all.

Root Cause

The problematic code block:

cachefile = os.path.join(cachedir, "annots.pkl")
...
if not os.path.isfile(cachefile):
    recs = {...}  # parsed from XML
    pickle.dump(recs, f)
else:
    with open(cachefile, "rb") as f:
        recs = pickle.load(f)   # <-- trusted forever, never checked

recs is then indexed directly per-image with no existence check:

R = [obj for obj in recs[imagename] if obj["name"] == classname]

If imagename isn't a key (because the cache predates that image), this raises KeyError: '<imagename>' instead of falling back to parsing the XML.

Environment

• YOLOX: main branch (voc_eval.py / voc.py unchanged as of this report)
• OS: Windows 11, Python 3.12, PyTorch 2.13
• Custom single-class VOC-format dataset, tiled and re-tiled across multiple labeling sessions

Steps to Reproduce

1. Train on dataset A long enough for eval to trigger at least once (writes annots.pkl covering A's images)
2. Add or relabel images, or regenerate a tiled dataset, changing the image set
3. Resume training without deleting annotations_cache/
4. Observe KeyError: '<imagename>' in voc_eval.py on the next eval

Suggested Fix

Make the cache self-healing instead of blindly trusted:

else:
    with open(cachefile, "rb") as f:
        recs = pickle.load(f)
    missing = [name for name in imagenames if name not in recs]
    if missing:
        print(f"Annotation cache {cachefile} is stale "
              f"({len(missing)} of {len(imagenames)} image(s) missing) - rebuilding.")
        recs = {}
        for i, imagename in enumerate(imagenames):
            recs[imagename] = parse_rec(annopath.format(imagename))
        with open(cachefile, "wb") as f:
            pickle.dump(recs, f)

Verified against a real dataset that hit this exact problem: eval that previously crashed with KeyError now rebuilds the cache and completes normally, and stays fast (no rebuild) on subsequent evals once the cache is back in sync with the dataset.

Start in yolox/evaluators/voc_eval.py at voc_eval() and inspect how annots.pkl is loaded and indexed. Reproduce the issue by evaluating dataset A, changing its image set, and evaluating again without removing annotations_cache/. Done when a stale cache rebuilds and the subsequent evaluation completes without KeyError while unchanged data continues to reuse the cache.