allenai / allenai/OLMo-core

convert_checkpoint_to_hf.py ignores -t/--tokenizer when sibling tokenizer/ directory exists

Open
#609 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.5k
Forks
315
Avg merge
1d 9h
Merged PRs (30d)
11

Description

## Bug

In `src/examples/huggingface/convert_checkpoint_to_hf.py` (the non-hybrid converter), the `-t`/`--tokenizer` CLI flag is ignored when a sibling `tokenizer/` directory exists next to the checkpoint.

**Lines 143-154:**
```python
tokenizer_path = Path(original_checkpoint_path).parent / "tokenizer"
huggingface_tokenizer = None
if tokenizer_path.exists():
log.info(f"Saving preexisting tokenizer from {tokenizer_path}")
huggingface_tokenizer = AutoTokenizer.from_pretrained(tokenizer_path)
huggingface_tokenizer.save_pretrained(output_path)
...
else:
tokenizer_id = tokenizer_id or tokenizer_config.identifier
...
```

The `tokenizer_id` (from `-t`) is only consulted in the `else` branch, so if the checkpoint was saved with a tokenizer directory (common for SFT checkpoints), the explicit `-t` override is silently ignored.

**Expected behavior:** `-t` should take precedence over the sibling `tokenizer/` directory when explicitly provided.

**Note:** The hybrid converter (`convert_checkpoint_to_hf_hybrid.py`) does not have this issue — it always respects `-t`.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.