bytedance / bytedance/dplm

Unclear config for released DPLM2 tokenizer checkpoint

Open
#52 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
345
Forks
52
PR merge metrics
No merged PRs in 30d

Description

I am trying to understand and reproduce the tokenizer used in DPLM2, but I found that the tokenizer training setup is currently unclear.

Specifically:

- The repository includes the config file
`configs/experiment/structok/structok_lfq_8k_pdb_swissprot_c512.yaml`

- However, the repo does not explicitly confirm that this is the exact configuration used to train the released tokenizer checkpoint.

I attempted to train a tokenizer based on the available config, but the resulting tokenizer does not match the results of the released tokenizer checkpoint. This makes it difficult to verify whether the discrepancy comes from missing code, missing configs, or differences in the actual training setup.

Could you please clarify:

1. Whether `structok_lfq_8k_pdb_swissprot_c512.yaml` corresponds to the tokenizer used in the released checkpoint, or if it differs in any important way?

2. Whether there are additional (currently undocumented) steps or settings required to reproduce the tokenizer?

3. Whether you could disclose the tokenizer training time and number of GPUs, which would be very helpful for understanding reproducibility and computational requirements?

Thanks again for your work and for any clarification you can provide.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading configs/experiment/structok/structok_lfq_8k_pdb_swissprot_c512.yaml and comparing its settings with the released tokenizer checkpoint setup described in the repository. Done means documenting whether this config is exact, identifying any additional training steps or settings, and recording the tokenizer training time and GPU count if available.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
documentation, machine-learning
Issue type
Documentation
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.