Alberto-Codes / Alberto-Codes/vramfit
docs: map README and card disagree on the tier-1/tier-2 corpus
- Dominant language
- Python
- Stars
- 1
- Forks
- 1
- Avg merge
- 5h 57m
- Merged PRs (30d)
- 110
Description
## Two published documents name different corpora for the same numbers
`publication/gemma-4-31b-sensitivity-maps/README.md` says:
> The evaluation tiers ran the packed model on held-out WikiText-2
> test text, never on either file.
`publication/gemma-4-31b-fit24gib/README.md` says the opposite, for
the same pack and the same tier-1 and tier-2 figures:
> Measured on the framed calibration corpus: 357 blocks of public-
> domain text under the Gemma channel frame, 182,404 tokens, 356
> chunks at n_ctx 512. This is the pack's measurement frame — the
> same corpus the importance matrix consumed — not held-out text.
## The card is the correct one, measured
Running the published pack on WikiText-2 test text at the card's
window, b10362 CUDA on a rented H100 SXM:
| Model | Corpus | Mean PPL |
|---|---|---|
| `gemma-4-31B-it-fit24gib.gguf` | WikiText-2 test, 60 chunks | 1382.01 |
| BF16 reference | WikiText-2 test, 564 chunks | 1841.13 |
| BF16 reference | framed calibration, 347 chunks | 78.81 |
The card records 37.4552 for that pack. WikiText-2 cannot be the
corpus behind it. The map dataset's own text explains why: Gemma 4
31B IT-QAT "prices raw prose at a perplexity near 3,000 and the same
prose inside its own channel at 26 to 75".
## Cost
The sentence sent a measurement run to the wrong corpus before the
card's own corpus line corrected it. A reader reproducing the card's
tier-1 or tier-2 number would hit the same wall.
## Fix
Correct the map README's sentence to name the framed calibration
corpus for tiers 1 and 2, and keep the held-out claim where it
belongs — the benchmark table, which does run held-out splits.
Found while measuring asymmetric KV cache settings for #424.
Contributor guide
Research direction
Open publication/gemma-4-31b-sensitivity-maps/README.md and find the sentence describing the evaluation tiers. Compare it with publication/gemma-4-31b-fit24gib/README.md and the benchmark table context. Done means tiers 1 and 2 name the framed calibration corpus, while the held-out claim remains only where the benchmark table applies.
Written by the indexing model from the issue text.
Assessment
- Domain
- documentation
- Issue type
- Documentation
- Difficulty
- 1/5
- Estimated time
- Under an hour
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 92/100