apache / apache/opendal

`OpendalStore` double-prefixing when used with DataFusion's object store registry

Open
#7,786 2 comments 1 reaction 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
5.4k
Forks
825
Avg merge
1d 14m
Merged PRs (30d)
127

Description

### Problem

When wrapping an `OpendalStore` in a DataFusion session context, paths get double-prefixed, causing file lookups to fail.

**Minimal reproduction:**

```rust
let op = Operator::from_uri("hf://buckets/lhoestq/datasets")?;
let store = OpendalStore::new(op);

let ctx = SessionContext::new();
let url = Url::parse("hf://buckets/lhoestq/datasets")?;
ctx.register_object_store(&url, store);

// Fails — paths are double-prefixed
ctx.read_parquet("hf://buckets/lhoestq/datasets/...").await?;
```

**What happens:** DataFusion's `DefaultObjectStoreRegistry` matches on **scheme + authority only** (`hf://buckets`), so a query against `hf://buckets/lhoestq/datasets/prompts.chat/data/file.parquet` passes the **full** path `lhoestq/datasets/prompts.chat/data/file.parquet` to the store. But the `Operator` already has root `/lhoestq/datasets/`, so `OpendalStore` concatenates:

```
root + path = /lhoestq/datasets/ + lhoestq/datasets/prompts.chat/data/file.parquet
= /lhoestq/datasets/lhoestq/datasets/prompts.chat/data/file.parquet ❌
```

The only workaround I found is writing a custom `ObjectStore` wrapper that strips the registered prefix before passing operations to `OpendalStore`, then restores the full path in metadata so DataFusion's `contains()` check succeeds.

---

Technical deep-dive (click to expand)

### The full path transformation chain

**1. Registration key extraction** — `DefaultObjectStoreRegistry::register_store()` calls `get_url_key()`:

```rust
fn get_url_key(url: &Url) -> String {
format!("{}://{}", url.scheme(), &url[url::Position::BeforeHost..url::Position::AfterPort])
}
```

This means `hf://buckets/lhoestq/datasets` and `hf://buckets` both produce the key `"hf://buckets"`. There's no path component in the key.

**2. Store lookup** — When `read_parquet("hf://buckets/lhoestq/datasets/...")` is called, DataFusion calls `registry.get_store(url)` → `get_url_key("hf://buckets/lhoestq/datasets")` → `"hf://buckets"` → matches our registered store. The full URL is then passed to the store's methods.

**3. Listing path** — In `ListingTableUrl::list_prefixed_files()`:
- The URL is parsed into a `ListingTableUrl` with `prefix = Path::from_url_path("/lhoestq/datasets/prompts.chat/data/...")`
- The prefix is normalized to `lhoestq/datasets/prompts.chat/data/...` (leading `/` stripped)
- If it looks like a single file (no trailing `/`), DataFusion calls `store.head(&full_prefix)` directly with `lhoestq/datasets/prompts.chat/data/...`
- If it looks like a directory, DataFusion calls `store.list(Some(&prefix))` with the same path

**4. `OpendalStore::get_opts()` path handling** — The store receives location `lhoestq/datasets/prompts.chat/data/...` and passes it directly to the OpenDAL operator:

```rust
async fn get_opts(&self, location: &Path, options: GetOptions) -> ... {
let raw_location = percent_decode_path(location.as_ref());
// raw_location = "lhoestq/datasets/prompts.chat/data/..."
// Operator internally does: root + path
// = "/lhoestq/datasets/" + "lhoestq/datasets/prompts.chat/data/..."
// = "/lhoestq/datasets/lhoestq/datasets/prompts.chat/data/..." ❌
}
```

There is no way to tell `OpendalStore` what prefix was registered in the outer context. It just blindly concatenates `operator.root + incoming_path`.

### Why this is tricky

1. DataFusion's registry key is `scheme://authority` (no path) — there's no way to register at a more specific path.
2. `OpendalStore` has no concept of a "registered prefix" — it just concatenates whatever path it receives with the operator's root.

### How to fix it

Is it a possible solution to use an empty root in the HF implementation @kszucs @Xuanwo ?

Or let me know if you have other ideas or if this issue should be in datafusion instead.

### Environment

- `opendal`: 0.57.x
- `object_store_opendal`: 0.57.x
- `object_store`: 0.13.x
- `datafusion`: 54.x

Contributor guide

Open the contributing guide

Research direction

Start with the minimal reproduction in the issue, then trace OpendalStore::get_opts alongside DataFusion's DefaultObjectStoreRegistry and ListingTableUrl path handling. Determine how the registered prefix should be represented in the HF implementation without changing unrelated path behavior. Done means parquet lookup through the registry no longer double-prefixes the operator root.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.