deepmodeling / deepmodeling/unimol_tools

[Code scan] Make UniMol tokenizer save and load self-contained

Open
#33 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
34
Forks
9
PR merge metrics
No merged PRs in 30d

Description

This issue is a result of a Codex global repository scan.

## Summary
UnimolTokenizer.save_pretrained writes tokenizer_config.json and mol.dict.txt, but from_pretrained requires UnimolConfig.from_pretrained. A tokenizer-only saved directory lacks config.json, so it cannot be reloaded as a standalone tokenizer. The implementation also bypasses Transformers vocab-file resolution by directly joining local paths.

## Code references
https://github.com/deepmodeling/unimol_tools/blob/4596596aa8f73eb462d5cc5a921d79966d0465da/unimol_hf/tokenization_unimol.py#L122-L158

## Impact
Users who save just the tokenizer, or load a tokenizer from a cached/remote Transformers layout, can hit config-loading failures even though the saved tokenizer directory contains the dictionary and tokenizer metadata.

## Suggested fix
Let PreTrainedTokenizer handle save/load metadata where possible, read tokenizer_config.json directly, accept resolved vocab_file or dict_path, and treat model config as optional defaults rather than a hard requirement.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.