tensorflow / tensorflow/model-optimization
Non-uniform sparsity layer-wise with PolynomialDecay scheduler
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.6k
- Forks
- 349
- Avg merge
- 3d 2h
- Merged PRs (30d)
- 1
Description
System information
- TensorFlow version (you are using): 2.9.1
- Are you willing to contribute it (Yes/No): Yes
Motivation
Instead of assuming that the initial and final sparsity are uniform across layers, is it possible to add a feature where the user can feed in either a custom sparsity map or a sparsity distribution generated using ERK (like in @evcu 's Rigl codebase - https://github.com/google-research/rigl/blob/master/rigl/sparse_utils.py)
@evcu 's experiments have shown ERK to work better and in general polynomial scheduler also seems to work better, so incorporating that with the tfmot call would be very helpful.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Begin by examining the existing tfmot PolynomialDecay pruning path and compare its sparsity handling with rigl/sparse_utils.py in the linked RigL codebase. Define how a custom sparsity map or ERK-generated distribution would enter the API and verify that sparsity can vary by layer while the scheduler still behaves as requested.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, tensorflow
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100