Caching support for long-running transformations
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 308
- Avg merge
- 3d 9h
- Merged PRs (30d)
- 14
Description
To speed up the compilation process for large models or large layers, it would make sense to have a caching mechanism for long-running transformations. The cached outputs would be persistent and get reused when appropriate when a transform is called. The cache generation/reuse should be optional.
The idea would be to generate a hash for all relevant input data for a transform for a particular node (including node attributes, parameter tensors, quantization annotations...) and use that hash as the cache key for the output products in a folder in a persistent location. Later on, if the same transformation is executed on a node with the same hash, the cached outputs can be reused by copying from the persistent cache folder into a new folder.
For NodeLocalTransform the hashing is relatively straightforward as only the node itself is passed to the transformation. For others, it's hard to generalize, and best considered on a case-by-case basis that gives the most execution time benefits for the use cases we have.
@quetric @Tobi-Alonso do you have any suggestions for which transforms to start with to get the most benefit, or any other comments?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the transformation entry points, with NodeLocalTransform as the clearest initial case. Determine which node data must contribute to a cache key, where persistent outputs would live, and how caching would be enabled or disabled. Done means a defined, optional cache design that safely reuses matching outputs; no files or tests are named in the issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- compilers, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100