deepmodeling / deepmodeling/unimol_tools
[Code scan] Make UniMol tokenizer save and load self-contained
- Dominant language
- Python
- Stars
- 34
- Forks
- 9
- PR merge metrics
- No merged PRs in 30d
Description
This issue is a result of a Codex global repository scan.
## Summary
UnimolTokenizer.save_pretrained writes tokenizer_config.json and mol.dict.txt, but from_pretrained requires UnimolConfig.from_pretrained. A tokenizer-only saved directory lacks config.json, so it cannot be reloaded as a standalone tokenizer. The implementation also bypasses Transformers vocab-file resolution by directly joining local paths.
## Code references
https://github.com/deepmodeling/unimol_tools/blob/4596596aa8f73eb462d5cc5a921d79966d0465da/unimol_hf/tokenization_unimol.py#L122-L158
## Impact
Users who save just the tokenizer, or load a tokenizer from a cached/remote Transformers layout, can hit config-loading failures even though the saved tokenizer directory contains the dictionary and tokenizer metadata.
## Suggested fix
Let PreTrainedTokenizer handle save/load metadata where possible, read tokenizer_config.json directly, accept resolved vocab_file or dict_path, and treat model config as optional defaults rather than a hard requirement.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.