Issue running parametric umap on large data - proto limit
Open
@timsainb is already working on this.
Since Mar 11, 2021.
- Dominant language
- Python
- Stars
- 8.3k
- Forks
- 871
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 5
Description
I'm trying to run a basic parametric UMAP embedding on my dataset. Each image is (193, 229, 193) so flattened out is 8,530,021 dimensions.
This is the error when trying to run `embedding = embedder.fit_transform(train_images)`.
```
raise ValueError(
ValueError: Tried to convert 'params' to a tensor and failed. Error: Cannot create a tensor proto whose content is larger than 2GB.
```
It seems like this is just a limitation of tensorflow so I am not sure what can be done about this on your end. But I was wondering if anyone has faced something similar with large datasets and if there was a workaround for this issue?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.