epfl-dlab / epfl-dlab/zip2zip-compression

`CompressionConfig` accepts out-of-vocab `disabled_ids` without validation

Open Beginner friendly
#7 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
2
Forks
2
PR merge metrics
No merged PRs in 30d

Description

`disabled_ids` are collected into a set with no range check against
`initial_vocab_size`. Ids that can never occur (e.g. Llama ids 128000+ with a 32k
model) are silently accepted and protect nothing — this turned a wrong tokenizer
default upstream into silently-unprotected special tokens. *Fix:* error (or warn) when
any disabled id is `>= initial_vocab_size`. (`src/config.rs:39-48` — `new()`, ids
collected at 46-47 with no range check; `src/codec.rs`)

Contributor guide

No contributing guide indexed for this repository

Research direction

Read src/config.rs:39-48, especially CompressionConfig::new() and the disabled_ids collection, then inspect the related path in src/codec.rs. Verify how ids are compared with initial_vocab_size and define completion as out-of-vocabulary disabled ids producing the issue's intended error or warning instead of being silently accepted.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
tooling
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
70/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.