huggingface / huggingface/xet-core

`XetSession::new_upload_commit().build()` scans the entire shard cache (~4s) on every call

Open
#853 4 comments 1 reaction 0 assignees View on GitHub
Dominant language
Rust
Stars
592
Forks
102
Avg merge
5d 8h
Merged PRs (30d)
9

Description

## Description

Every call to `XetSession::new_upload_commit().build().await` triggers a full scan and in-memory indexing of the local shard cache, regardless of whether a prior session already performed that work. On a machine with a populated cache (708 `.mdb` files, ~90 MB), this consistently takes ~4 seconds per upload session.

## Call chain

```
XetUploadCommitBuilder::build().await
→ FileUploadSession::new()
→ SessionShardInterface::new()
→ ShardFileManager::new_in_cache_directory()
→ refresh_shard_dir()
→ load_managed_directory() // 2.15s — reads/parses all .mdb file headers
→ register_shards() // 2.0s — reads all chunk hashes, builds in-memory lookup
```

## Root cause

`ShardFileManager` is cached per `XetContext` (via `XetCommon::runtime_cache`). However, `XetSessionBuilder::build()` always constructs a brand-new `XetContext` — there is no mechanism to pass in an existing one. As a result, every `XetSession` starts with an empty shard-file-manager cache and must re-scan the full cache directory from disk.

The legacy Python-binding API (`upload_files` / `upload_bytes` in `hf_xet::legacy`) avoids this by holding a process-global `XetContext` in a `lazy_static` (`MULTITHREADED_RUNTIME`). The cache directory is scanned once per process lifetime. Any caller using `XetSessionBuilder` directly — including OpenDAL's HuggingFace service — pays the full scan cost on every upload.

## Observed via OpenDAL integration tests

Added some timings to xet-core:

```
❯ RUST_LOG=opendal_service_hf=debug,xet_data=debug OPENDAL_TEST=hf OPENDAL_HF_REPO_TYPE=bucket OPENDAL_HF_REPO_ID=kszucs/opendal RUST_TEST_THREADS=1 cargo test -p opendal --features services-hf,tests "behavior::test_write_only"
Finished `test` profile [unoptimized + debuginfo] target(s) in 0.25s
Running unittests src/lib.rs (target/debug/deps/opendal-90ceca890413e75f)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

Running tests/behavior/main.rs (target/debug/deps/behavior-338974d2e605e604)

running 1 test
2026-05-26T11:12:10.146695+02:00 DEBUG opendal_service_hf::core: core.rs:254 new_upload_commit (sync) took 30.875µs
[xet-timing] create_remote_client took 1.464291ms
[xet-timing] merge_shards_background spawn took 418.667µs
[xet-timing] load_managed_directory (710 files) took 2.038852416s
[xet-timing] shard "9b477be08c08de5844b3d67c28af0e5363c6850c6522951ce2b09ee52a3dccbb.mdb" (2976 bytes): read_all_truncated_hashes + bookkeeper update took 213.917µs
[xet-timing] shard "d8b0c4fb3f44f027bb9b6b8ad99b70a101ba80a85b4a48600ace439d214a620b.mdb" (3872 bytes): read_all_truncated_hashes + bookkeeper update took 17.125µs
[xet-timing] shard "d8eafa4b2147ed3fe3ecf10455ffd59e07783e3516ac4dd3a02d1d6dec7dc002.mdb" (4128 bytes): read_all_truncated_hashes + bookkeeper update took 14.666µs
[xet-timing] shard "940d29b15d2538298905e95ae3609545920762eb547d6015cb6c05921e94844c.mdb" (2016 bytes): read_all_truncated_hashes + bookkeeper update took 13.584µs
...
[xet-timing] register_shards (710 shards) took 1.877259125s
[xet-timing] ShardFileManager::new_in_cache_directory took 3.9170355s
[xet-timing] ShardFileManager::new_in_session_directory took 22.292µs
[xet-timing] shard_merge_jh.await took 31.25µs
[xet-timing] SessionShardInterface::new took 3.919933625s
2026-05-26T11:12:14.414164+02:00 DEBUG opendal_service_hf::core: core.rs:265 upload_commit.build() took 4.263337292s
2026-05-26T11:12:14.414193+02:00 DEBUG opendal_service_hf::writer: writer.rs:47 xet_upload_commit took 4.267583917s
2026-05-26T11:12:14.414360+02:00 DEBUG opendal_service_hf::writer: writer.rs:57 upload_stream took 157.75µs
[xet-timing] shard "c7c46df92f3e5784161b9ed2aeca2a7d3ca2427c0f247c1a9c0fdd74b0b15442.mdb" (2784 bytes): read_all_truncated_hashes + bookkeeper update took 146.25µs
[xet-timing] shard "af5685573d6e245ba3b86dcdb98e30106924fc43de2fa3b921853b06afa1e350.mdb" (2784 bytes): read_all_truncated_hashes + bookkeeper update took 156.667µs
test behavior::test_write_only ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 91 filtered out; finished in 7.14s
```

Contributor guide

Open the contributing guide

Research direction

Start at XetSessionBuilder::build(), then trace XetUploadCommitBuilder::build() through FileUploadSession::new(), SessionShardInterface::new(), and ShardFileManager::new_in_cache_directory(). Compare this with the legacy hf_xet::legacy MULTITHREADED_RUNTIME path and its shared XetContext. Done means repeated upload-session builds reuse the shard cache rather than rescanning all .mdb files; verify with the OpenDAL behavior::test_write_only integration test and timing logs.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.