mlcommons / mlcommons/algorithmic-efficiency
LM1B: Jax
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 425
- Forks
- 78
- PR merge metrics
- No merged PRs in 30d
Description
Workload
LM1B
Task
Transformer language model training, resembling https://github.com/google/flax/tree/main/examples/lm1b .
Dataset
LM1B dataset. https://www.tensorflow.org/datasets/catalog/lm1b
Model
decoder-only Transformer, encoder+decoder based Transformer.
Reference Implementation
https://github.com/google/flax/tree/main/examples/lm1b
ToDo
- Implement data input pipeline
- Document specific dataset version in workload-specific README
- Add model
- Document model in workload-specific README
- Provide sample submission (and sample tuning search space)
- Document results of sample submission in workload-specific README (how long did it take to reach the target performance)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the Google Flax LM1B reference implementation and the linked TensorFlow Datasets LM1B specification. Define the data pipeline, decoder-only and encoder-decoder Transformer models, and workload-specific README documentation. Done means the workload runs, includes a sample submission and tuning search space, and records how long it takes to reach the target performance.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, tensorflow
- Domain
- data, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100