apache / apache/datafusion-comet
Object store cache key drops the ABFS container, so two containers in one storage account share a store instance
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
## Describe the bug
`prepare_object_store_with_configs` in `native/core/src/parquet/parquet_support.rs` caches object store instances under `(url_key, config_hash)`, where `url_key` is built from:
```rust
let url_key = format!(
"{}://{}",
scheme,
&url[url::Position::BeforeHost..url::Position::AfterPort],
);
```
`Position::BeforeHost` starts *after* the URL userinfo. For ABFS, the container is the userinfo, not the host:
```
abfss://container@account.dfs.core.windows.net/path/to/file.parquet
```
So `abfss://c1@acct.dfs.core.windows.net/...` and `abfss://c2@acct.dfs.core.windows.net/...` both produce the key `abfss://acct.dfs.core.windows.net`. `config_hash` is computed over the per-session Hadoop config map, which does not vary by container, so it does not disambiguate them either.
Meanwhile `MicrosoftAzureBuilder::parse_url` bakes the container into the store instance from the URL userinfo, and the `Path` handed to that store is container-relative (the bucket/container is stripped by `Path::from_url_path(url.path())` in `PhysicalPlanner::get_partitioned_files`).
The consequence: within one executor process, reading `abfss://c2@acct/data/f.parquet` after `abfss://c1@acct/data/f.parquet` should hit the cached `c1` store and read **c1's data**.
## Steps to reproduce
Not yet reproduced. This was found by code reading during review of an unrelated change. Confirming it needs an Azure storage account with two containers holding same-named paths, read from a single executor in one application.
## Expected behavior
Each ABFS container resolves to its own object store instance, and a read from one container never returns another container's data.
## Additional context
- S3, GCS, and HDFS are unaffected: their bucket/host lives in the URL host component, which `url_key` does keep.
- The same key derivation is now also used to key a shared Parquet metadata cache, where the impact is narrower (entries are additionally validated against file size and modification time). That is documented as a known limitation there; this issue is about the store cache itself, where no such validation exists.
- The object store cache is process-wide and unbounded by design, so a wrong entry persists for the executor's lifetime.
Contributor guide
Assessment
This issue has not been assessed yet.