developmentseed / developmentseed/obstore
fsspec integration fails to read Azure directory dataset
- Dominant language
- Python
- Stars
- 810
- Forks
- 42
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 6
Description
### Problem:
Attempts to read the dataset using `pyarrow.dataset.dataset(path, filesystem=obstore_fs)` consistently fail. Debugging revealed that the underlying `obstore.fsspec.FsspecStore` instance (`obstore_fs`) fails to access the target path, even for simple operations like `obstore_fs.info(path)`, typically raising a `FileNotFoundError`.
### Context:
* **STAC Collection:** `landsat-c2-l2` on Planetary Computer STAC (`https://planetarycomputer.microsoft.com/api/stac/v1/`)
* **STAC Asset:** `geoparquet-items`
* **Asset Details:**
* `href`: `abfs://items/landsat-c2-l2.parquet`
* `table:storage_options`: `{'account_name': 'pcstacitems'}`
* **Target Path Type:** The path `abfs://items/landsat-c2-l2.parquet` points to a **directory** containing the partitioned Parquet dataset, not a single file.
### Steps Taken & Findings:
1. Retrieved STAC asset using `pystac-client`.
2. Initialized `obstore.auth.planetary_computer.PlanetaryComputerCredentialProvider` correctly using `url=asset.href` and `account_name='pcstacitems'`.
3. Attempted to initialize `obstore.fsspec.FsspecStore` with various configurations:
* Using `protocol='abfs'` and the credential provider.
* Using `protocol='https'` (with a constructed HTTPS URL) and the provider.
* Explicitly passing `config={'account_name': 'pcstacitems', 'container_name': 'items'}` along with the provider for both `abfs` and `https` protocols.
4. In all `obstore.fsspec` configurations, attempts to access the path (e.g., `abfs://items/landsat-c2-l2.parquet`) via `fs.info()` failed with `FileNotFoundError` (referencing either `landsat-c2-l2.parquet` or `items/landsat-c2-l2.parquet`). `fs.info()` works for files but seems to fail for directories in this setup.
5. Consequently, `pyarrow.dataset.dataset(path, filesystem=obstore_fs)` also failed, presumably because it couldn't access/list the directory contents via the `obstore` fsspec layer.
6. **Successful Native `fsspec` Test:** A separate test using native `fsspec` and `adlfs` was performed:
* Obtained the SAS token directly by calling the initialized `PlanetaryComputerCredentialProvider` instance.
* Created a native filesystem: `native_fs = fsspec.filesystem("abfs", account_name='pcstacitems', sas_token=...)`.
* Accessed the directory info using the relative path: `native_fs.info('items/landsat-c2-l2.parquet')`. **This succeeded** and correctly identified the path as a **directory**.
* Opened the dataset using `pyarrow.dataset.dataset('items/landsat-c2-l2.parquet', filesystem=native_fs)`. **This also succeeded.**
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.