Alberto-Codes / Alberto-Codes/vramfit

docs: map README and card disagree on the tier-1/tier-2 corpus

Open Beginner friendly
#577 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1
Forks
1
Avg merge
5h 57m
Merged PRs (30d)
110

Description

## Two published documents name different corpora for the same numbers

`publication/gemma-4-31b-sensitivity-maps/README.md` says:

> The evaluation tiers ran the packed model on held-out WikiText-2
> test text, never on either file.

`publication/gemma-4-31b-fit24gib/README.md` says the opposite, for
the same pack and the same tier-1 and tier-2 figures:

> Measured on the framed calibration corpus: 357 blocks of public-
> domain text under the Gemma channel frame, 182,404 tokens, 356
> chunks at n_ctx 512. This is the pack's measurement frame — the
> same corpus the importance matrix consumed — not held-out text.

## The card is the correct one, measured

Running the published pack on WikiText-2 test text at the card's
window, b10362 CUDA on a rented H100 SXM:

| Model | Corpus | Mean PPL |
|---|---|---|
| `gemma-4-31B-it-fit24gib.gguf` | WikiText-2 test, 60 chunks | 1382.01 |
| BF16 reference | WikiText-2 test, 564 chunks | 1841.13 |
| BF16 reference | framed calibration, 347 chunks | 78.81 |

The card records 37.4552 for that pack. WikiText-2 cannot be the
corpus behind it. The map dataset's own text explains why: Gemma 4
31B IT-QAT "prices raw prose at a perplexity near 3,000 and the same
prose inside its own channel at 26 to 75".

## Cost

The sentence sent a measurement run to the wrong corpus before the
card's own corpus line corrected it. A reader reproducing the card's
tier-1 or tier-2 number would hit the same wall.

## Fix

Correct the map README's sentence to name the framed calibration
corpus for tiers 1 and 2, and keep the held-out claim where it
belongs — the benchmark table, which does run held-out splits.

Found while measuring asymmetric KV cache settings for #424.

Contributor guide

Open the contributing guide

Research direction

Open publication/gemma-4-31b-sensitivity-maps/README.md and find the sentence describing the evaluation tiers. Compare it with publication/gemma-4-31b-fit24gib/README.md and the benchmark table context. Done means tiers 1 and 2 name the framed calibration corpus, while the held-out claim remains only where the benchmark table applies.

Written by the indexing model from the issue text.

Assessment

Domain
documentation
Issue type
Documentation
Difficulty
1/5
Estimated time
Under an hour
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
92/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.