epfl-dlab / epfl-dlab/zip2zip-compression
`CompressionConfig` accepts out-of-vocab `disabled_ids` without validation
- Dominant language
- Rust
- Stars
- 2
- Forks
- 2
- PR merge metrics
- No merged PRs in 30d
Description
`disabled_ids` are collected into a set with no range check against
`initial_vocab_size`. Ids that can never occur (e.g. Llama ids 128000+ with a 32k
model) are silently accepted and protect nothing — this turned a wrong tokenizer
default upstream into silently-unprotected special tokens. *Fix:* error (or warn) when
any disabled id is `>= initial_vocab_size`. (`src/config.rs:39-48` — `new()`, ids
collected at 46-47 with no range check; `src/codec.rs`)
Contributor guide
No contributing guide indexed for this repository
Research direction
Read src/config.rs:39-48, especially CompressionConfig::new() and the disabled_ids collection, then inspect the related path in src/codec.rs. Verify how ids are compared with initial_vocab_size and define completion as out-of-vocabulary disabled ids producing the issue's intended error or warning instead of being silently accepted.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- tooling
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 70/100