Resolved to no files error. using dd.read_csv and a globstring
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 398
- Forks
- 181
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 36
Description
Hi!
dask 2.8.0, gcsfs 0.4.0
I have the following error when trying to use a glob string in a file path:
---------------------------------------------------------------------------
OSError Traceback (most recent call last)
<ipython-input-14-559125deb7ec> in <module>
5 ) and clusterid !=32003
6 ''',
----> 7 compute=True)
<ipython-input-10-3ef47a2c7f4f> in df_from_bq(query, table, compute, output, dtype)
60 pass
61
---> 62 df = dd.read_csv(gs_path+'{0}'.format(destination_file), storage_options={'token': key}, dtype=dtype, low_memory=False)
63
64 if compute==True and output==True:
~\AppData\Local\Continuum\anaconda3\lib\site-packages\dask\dataframe\io\csv.py in read(urlpath, blocksize, collection, lineterminator, compression, sample, enforce, assume_missing, storage_options, include_path_column, **kwargs)
576 storage_options=storage_options,
577 include_path_column=include_path_column,
--> 578 **kwargs
579 )
580
~\AppData\Local\Continuum\anaconda3\lib\site-packages\dask\dataframe\io\csv.py in read_pandas(reader, urlpath, blocksize, collection, lineterminator, compression, sample, enforce, assume_missing, storage_options, include_path_column, **kwargs)
403 compression=compression,
404 include_path=include_path_column,
--> 405 **(storage_options or {})
406 )
407
~\AppData\Local\Continuum\anaconda3\lib\site-packages\dask\bytes\core.py in read_bytes(urlpath, delimiter, not_zero, blocksize, sample, compression, include_path, **kwargs)
94
95 if len(paths) == 0:
---> 96 raise IOError("%s resolved to no files" % urlpath)
97
98 if blocksize is not None:
OSError: gs://BQ-Extracts/result_20191118_111323_*.csv resolved to no files
Everything works fine when I indicate a filepath without a globstring like this
gs://BQ-Extracts/result_20191118_111323_00000.csv however it fails with the abovementioned error with a globstring. Need to fix this issue as the code is used inside a big function
The issue appeared after upgrading gcsfs to the latest version. Everything works with globstring if I downgrade gcsfs to 0.3.0
UPD: Just a run the same query a few more time with gcsfs 0.3.0 and received the 'resolved to no files' error. Probably the newer version of dask or fsspec 0.6.0 cause the issues
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No repository file or test is named. Start by reproducing the reported gs:// glob with the listed dask, gcsfs, and fsspec versions, then compare it with a direct filepath and the older gcsfs version. Done means a matching glob resolves to the expected files consistently.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- gcp, python
- Domain
- cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100