developmentseed / developmentseed/obstore

FsspecStore glob fails with "Last-Modified Header missing from response"

Open
#438 3 comments 1 reaction 0 assignees View on GitHub
fsspec
Dominant language
Python
Stars
810
Forks
42
Avg merge
1d 15h
Merged PRs (30d)
6

Description

```
[nav] In [21]: from obstore.fsspec import FsspecStore
...: FsspecStore("gs").glob("gs://gcp-public-data-landsat/*.csv.gz")
---------------------------------------------------------------------------
GenericError Traceback (most recent call last)
Input In [21], in ()
1 from obstore.fsspec import FsspecStore
----> 2 FsspecStore("gs").glob("gs://gcp-public-data-landsat/*.csv.gz")

File ~/model312/.venv/lib/python3.13/site-packages/fsspec/asyn.py:118, in sync_wrapper..wrapper(*args, **kwargs)
115 @functools.wraps(func)
116 def wrapper(*args, **kwargs):
117 self = obj or args[0]
--> 118 return sync(self.loop, func, *args, **kwargs)

File ~/model312/.venv/lib/python3.13/site-packages/fsspec/asyn.py:103, in sync(loop, func, timeout, *args, **kwargs)
101 raise FSTimeoutError from return_result
102 elif isinstance(return_result, BaseException):
--> 103 raise return_result
104 else:
105 return return_result

File ~/model312/.venv/lib/python3.13/site-packages/fsspec/asyn.py:56, in _runner(event, coro, result, timeout)
54 coro = asyncio.wait_for(coro, timeout=timeout)
55 try:
---> 56 result[0] = await coro
57 except Exception as ex:
58 result[0] = ex

File ~/model312/.venv/lib/python3.13/site-packages/fsspec/asyn.py:804, in AsyncFileSystem._glob(self, path, maxdepth, **kwargs)
801 else:
802 depth = None
--> 804 allpaths = await self._find(
805 root, maxdepth=depth, withdirs=True, detail=True, **kwargs
806 )
808 pattern = glob_translate(path + ("/" if ends_with_sep else ""))
809 pattern = re.compile(pattern)

File ~/model312/.venv/lib/python3.13/site-packages/fsspec/asyn.py:846, in AsyncFileSystem._find(self, path, maxdepth, withdirs, **kwargs)
842 detail = kwargs.pop("detail", False)
844 # Add the root directory if withdirs is requested
845 # This is needed for posix glob compliance
--> 846 if withdirs and path != "" and await self._isdir(path):
847 out[path] = await self._info(path)
849 # async for?

File ~/model312/.venv/lib/python3.13/site-packages/fsspec/asyn.py:677, in AsyncFileSystem._isdir(self, path)
675 async def _isdir(self, path):
676 try:
--> 677 return (await self._info(path))["type"] == "directory"
678 except OSError:
679 return False

File ~/model312/.venv/lib/python3.13/site-packages/obstore/fsspec.py:488, in FsspecStore._info(self, path, **_kwargs)
485 store = self._construct_store(bucket)
487 try:
--> 488 head = await obs.head_async(store, path_no_bucket)
489 return {
490 # Required of `info`: (?)
491 "name": head["path"],
(...)
497 "version": head["version"],
498 }
499 except FileNotFoundError:

GenericError: Generic GCS error: Last-Modified Header missing from response

Debug source:
Generic {
store: "GCS",
source: Header {
source: MissingLastModified,
},
}
```
Looks like `glob` is trying to get access the root bucket "object" which isn't really there?
```
[ins] In [22]: %debug
> /Users/brett/model312/.venv/lib/python3.13/site-packages/obstore/fsspec.py(488)_info()
486
487 try:
--> 488 head = await obs.head_async(store, path_no_bucket)
489 return {
490 # Required of `info`: (?)

ipdb> store
GCSStore(bucket="gcp-public-data-landsat")
ipdb> path_no_bucket
''
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.