NVIDIA / NVIDIA/Megatron-LM

Determine need for tokenizer assets in MegatronLM checkpoints

Open
#6,005 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
17.9k
Forks
4.5k
Avg merge
4d 6h
Merged PRs (30d)
271

Description

Parent initiative: #2319

## Context

Megatron-Bridge checkpoints may include a `tokenizer/` directory containing tokenizer assets such as vocab and merges files.
It's unclear whether this should be included in the MegatronLM checkpoints.

## Task

Determine whether the tokenizer assets should be saved in MegatronLM checkpoints, and if so create a new issue with the necessary implementation requirements.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing parent issue #2319 and the checkpoint handling described in the issue. Compare whether Megatron-Bridge checkpoints’ tokenizer/ directory and its vocab or merges files are needed in MegatronLM checkpoints. Done means documenting the decision and, if required, creating a follow-up issue with implementation requirements.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.