Unclear config for released DPLM2 tokenizer checkpoint
- Dominant language
- Python
- Stars
- 345
- Forks
- 52
- PR merge metrics
- No merged PRs in 30d
Description
I am trying to understand and reproduce the tokenizer used in DPLM2, but I found that the tokenizer training setup is currently unclear.
Specifically:
- The repository includes the config file
`configs/experiment/structok/structok_lfq_8k_pdb_swissprot_c512.yaml`
- However, the repo does not explicitly confirm that this is the exact configuration used to train the released tokenizer checkpoint.
I attempted to train a tokenizer based on the available config, but the resulting tokenizer does not match the results of the released tokenizer checkpoint. This makes it difficult to verify whether the discrepancy comes from missing code, missing configs, or differences in the actual training setup.
Could you please clarify:
1. Whether `structok_lfq_8k_pdb_swissprot_c512.yaml` corresponds to the tokenizer used in the released checkpoint, or if it differs in any important way?
2. Whether there are additional (currently undocumented) steps or settings required to reproduce the tokenizer?
3. Whether you could disclose the tokenizer training time and number of GPUs, which would be very helpful for understanding reproducibility and computational requirements?
Thanks again for your work and for any clarification you can provide.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading configs/experiment/structok/structok_lfq_8k_pdb_swissprot_c512.yaml and comparing its settings with the released tokenizer checkpoint setup described in the repository. Done means documenting whether this config is exact, identifying any additional training steps or settings, and recording the tokenizer training time and GPU count if available.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- documentation, machine-learning
- Issue type
- Documentation
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100