graykode / graykode/matorage

A few questions for the usage

Open
#1 7 comments 1 reaction 0 assignees View on GitHub
question
Dominant language
Python
Stars
72
Forks
7
PR merge metrics
No merged PRs in 30d

Description

It's really fantastic! Thank you so much for sharing this project.
I had a quick test with `minio` docker process and confirmed it works really well as expected.
I'd like to ask a few questions about the usage:
- can I save any objects other than tensors, i.e. a tuple of tensors, a dict or a sparse tensors? In this case how to give the attributes?
- If I add some more data samples to an existing dataset (in the case that periodically data samples could be added so if I have to refresh all the datasets with the added samples), will it be okay to add to the dataset and save it?
- If I use this in a distributed training, will the each dataloader occupy the amount of dataset? For example, if I'd like to use this with PyTorch DDP, then each process will have its own `DataLoader` and will load the data samples during the training -- so I wonder the loaded dataset size will be multiple of the original dataset size per process or not.
- It looks like supporting only PyTorch and TensorFlow now, but how about the numpy array or matrix for scikit-learn or XGBoost? Can I store some numpy objects as well?

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.