PAIR-code / PAIR-code/tiny-transformers
Create a Gemma compatible transformer implementation
Open
Nobody has claimed this yet.
enhancement
help wanted
project:TTP
- Dominant language
- Jupyter Notebook
- Stars
- 24
- Forks
- 6
- PR merge metrics
- No merged PRs in 30d
Description
Different transformer implementations have variations (e.g. in positional encoding, where skip connections are, use of MQA, etc). Lets provide a Gemma standard implementation of transformers. This could be verified by being able to load and evaluate with a Gemma weights file.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files or tests are named. Start by reading the repository's existing transformer implementations and identify the entry point for loading model weights; compare their positional encoding, skip connections, and attention behavior with the Gemma standard. Done means the new implementation can load and evaluate a Gemma weights file.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- jupyter-notebook
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100