mlfoundations / mlfoundations/datacomp
Not able to push data to google cloud storage
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 792
- Forks
- 65
- PR merge metrics
- No merged PRs in 30d
Description
While trying to push data to the Google Cloud, I am getting a file not found error. Any help would be highly appreciated.
python download_upstream.py --scale medium --data_dir "gs://dataset/datacomp/" --thread_count 2
Downloading metadata to gs://dataset/datacomp/metadata...
Downloading (…)c76a589ef5d0.parquet: 100%|██████████████████████████████████████████████████████████████████| 122M/122M [00:00<00:00, 395MB/s]
Downloading (…)30fd0d497176.parquet: 100%|██████████████████████████████████████████████████████████████████| 122M/122M [00:00<00:00, 360MB/s]
.
.
.
Downloading (…)0edfcd0a6bc7.parquet: 100%|██████████████████████████████████████████████████████████████████| 121M/121M [00:00<00:00, 260MB/s]
Fetching 253 files: 100%|███████████████████████████████████████████████████████████████████████████████████| 253/253 [00:54<00:00, 4.67it/s]
Done downloading metadata.7.parquet: 26%|████████████████▉ | 31.5M/121M [00:00<00:00, 143MB/s]
Downloading images to gs://dataset/datacomp/shards██████████████████████████████████████████████ | 115M/121M [00:00<00:00, 307MB/s]
Starting the downloading of this file
Sharding file number 1 of 1 called dataset/datacomp/metadata
0it [00:08, ?it/s]
Traceback (most recent call last):
File "/home/mayank/datacomp/datacomp/download_upstream.py", line 218, in <module>
img2dataset.download(
File "/home/mayank/miniconda3/envs/datacomp/lib/python3.10/site-packages/img2dataset/main.py", line 232, in download
distributor_fn(
File "/home/mayank/miniconda3/envs/datacomp/lib/python3.10/site-packages/img2dataset/distributor.py", line 36, in multiprocessing_distributor
failed_shards = run(reader)
File "/home/mayank/miniconda3/envs/datacomp/lib/python3.10/site-packages/img2dataset/distributor.py", line 31, in run
for (status, row) in tqdm(process_pool.imap_unordered(downloader, gen)):
File "/home/mayank/miniconda3/envs/datacomp/lib/python3.10/site-packages/tqdm/std.py", line 1195, in __iter__
for obj in iterable:
File "/home/mayank/miniconda3/envs/datacomp/lib/python3.10/multiprocessing/pool.py", line 873, in next
raise value
FileNotFoundError: b/dataset/o/datacomp%2Fmetadata
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at download_upstream.py line 218 and reproduce the reported command with the gs:// data_dir. Trace how the path is passed to img2dataset and compare it with the FileNotFoundError path; done when the download completes successfully against Google Cloud Storage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- gcp, python
- Domain
- cloud, data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100