apache / apache/opendal

bug(services/hf): avoid extra HEAD probe for bucket reads

Open
#7,622 3 comments 2 reactions 0 assignees View on GitHub
bug releases-note/fix
Dominant language
Rust
Stars
5.4k
Forks
825
Avg merge
1d 14m
Merged PRs (30d)
127

Description

### Describe the bug

Users reported that reads from Hugging Face bucket through OpenDAL are slow and can hit rate limits.

For bucket repos, OpenDAL currently still calls `HfCore::maybe_xet_file()` before creating the reader. This sends a HEAD request to the resolve URL on every read to discover `X-Xet-Hash` and size metadata, even though buckets are expected to use XET.

Relevant code:

- `core/services/hf/src/reader.rs`: `HfReader::try_new()` calls `core.maybe_xet_file(path)` before reading.
- `core/services/hf/src/core.rs`: `maybe_xet_file()` issues a HEAD request and parses `X-Xet-Hash`.
- `core/services/hf/src/backend.rs`: bucket `stat()` uses the same probe.

### Steps to Reproduce

Read many objects or ranges from an HF bucket-backed OpenDAL operator.

The issue is easier to observe with workloads that repeatedly call read or range-read, such as parquet/object_store-style readers.

### Expected Behavior

HF bucket reads should avoid an extra per-read HEAD probe when the required XET metadata can be resolved through a bucket-native path, cached metadata, list/stat metadata, or another lower-request-count mechanism.

A normal read path should not double the number of HF-facing requests before data transfer starts.

### Additional Context

This is separate from #7577, which tracks HF dataset behavior-test timeouts on CI. This issue is specifically about HF bucket read request amplification and rate limiting.

Related broader API work: #5872 tracks returning metadata from read operations, which can help integrations avoid extra stat calls.

Contributor guide

Open the contributing guide

Research direction

Start in core/services/hf/src/reader.rs at HfReader::try_new(), then trace HfCore::maybe_xet_file() in core/services/hf/src/core.rs and the bucket stat path in core/services/hf/src/backend.rs. Run repeated bucket read and range-read workloads to measure HF-facing requests. Done means bucket reads avoid the extra per-read HEAD probe while still obtaining the required XET metadata and transferring data correctly.

Written by the indexing model from the issue text.

Assessment

Tech stack
huggingface, rust
Domain
cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.