tensorflow / tensorflow/privacy
High memory usage with IndexedSlices
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2k
- Forks
- 477
- Avg merge
- 22h 12m
- Merged PRs (30d)
- 1
Description
Hi,
I'm using an Embedding layer in my model.
When I train it with DPAdamOptimizer, I get a warning:
UserWarning: Converting sparse IndexedSlices to a dense Tensor with 163617900 elements. This may consume a large amount of memory.
Therefore, I have to use hardware with a huge amount of memory. Otherwise, I run into an OOM Error.
Is this intended behavior (as also 0-gradients have to be perturbed) or could this possibly be improved?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the reported training setup with an Embedding layer and DPAdamOptimizer, then inspect how the IndexedSlices warning and dense conversion arise. Determine whether the conversion is intended for perturbed zero gradients or causes avoidable memory use; done means the behavior is explained and, if appropriate, the OOM-producing behavior is addressed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, tensorflow
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100