PAIR-code / PAIR-code/tiny-transformers
Provide a visualization for the norms of different parts of a transformer model
Open
@AAlberich8 is already working on this.
Since Aug 29, 2024.
enhancement
project:TTP
- Dominant language
- Jupyter Notebook
- Stars
- 24
- Forks
- 6
- PR merge metrics
- No merged PRs in 30d
Description
A common debugging method is to monitor the norm of some part of a ML model's weight and activation space.
Goal:
- Create a visualization component for norms of a set of vectors and/or weights. Allow the norms to be tracked over time, like we do with loss/accuracy.
- The code and implementation should be reusable and clean.
Code pointers:
- NPM project for animated transformers: https://github.com/PAIR-code/tiny-transformers/tree/main/animated-transformer
- the current line-chart component (in case we can make this nicer, e.g. allow clicking on points to see exact values, user-changable colors, etc). https://github.com/PAIR-code/tiny-transformers/tree/main/animated-transformer/src/app/d3-line-chart
- The Angular subdirectory of that for the animated transformer tool vis: https://github.com/PAIR-code/tiny-transformers/tree/main/animated-transformer/src/app/animated-transformer
- HTML file for current plots during training: https://github.com/PAIR-code/tiny-transformers/blob/main/animated-transformer/src/app/animated-transformer/model-task-trainer/model-task-trainer.component.html#L57
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.