graykode / graykode/matorage

no metadata dir in a compressed bucket

Open
#18 6 comments 0 reactions 1 assignee Claimed by @graykode View on GitHub
bug good first issue
Dominant language
Python
Stars
72
Forks
7
PR merge metrics
No merged PRs in 30d

Description

Hi again,

Sorry for bothering you with several question and bug report, but this looks critical.
I made a compressed data bucket and it looks storing well, but when I retrieve the dataset, it has 0 len as follows:
```
Traceback (most recent call last):
File "/home/jinserk/.pyenv/versions/3.8.5/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/home/jinserk/kyu/kyumlm/mlmanager/torch/workers.py", line 202, in run
self.setup()
File "/home/jinserk/kyu/kyumlm/mlmanager/torch/workers.py", line 190, in setup
self.set_dataloaders()
File "/home/jinserk/kyu/kyumlm/mlmanager/torch/workers.py", line 134, in set_dataloaders
trainset, valset = self.set_datasets()
File "/home/jinserk/kyu/kyumlm/tddft/ann/workers.py", line 88, in set_datasets
print(dataset[0])
File "/home/jinserk/kyu/kyumlm/tddft/ann/dataset.py", line 35, in __getitem__
x = super().__getitem__(index)
File "/home/jinserk/.pyenv/versions/kyumlm/lib/python3.8/site-packages/matorage/data/torch/dataset.py", line 81, in __getitem__
return self._get_item_with_download(idx)
File "/home/jinserk/.pyenv/versions/kyumlm/lib/python3.8/site-packages/matorage/data/torch/dataset.py", line 89, in _get_item_with_download
_objectname, _relative_index = self._find_object(idx)
File "/home/jinserk/.pyenv/versions/kyumlm/lib/python3.8/site-packages/matorage/data/data.py", line 128, in _find_object
_key = self.end_indices[_key_idx]
IndexError: list index out of range
```
I've checked briefly, and found that the bucket has no `metadata` to read out the meta info of the dataset.
Can you fix this error? I have installed the latest master branch code.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.