huggingface / huggingface/xet-core
`XetSession::new_upload_commit().build()` scans the entire shard cache (~4s) on every call
- Dominant language
- Rust
- Stars
- 592
- Forks
- 102
- Avg merge
- 5d 8h
- Merged PRs (30d)
- 9
Description
## Description
Every call to `XetSession::new_upload_commit().build().await` triggers a full scan and in-memory indexing of the local shard cache, regardless of whether a prior session already performed that work. On a machine with a populated cache (708 `.mdb` files, ~90 MB), this consistently takes ~4 seconds per upload session.
## Call chain
```
XetUploadCommitBuilder::build().await
→ FileUploadSession::new()
→ SessionShardInterface::new()
→ ShardFileManager::new_in_cache_directory()
→ refresh_shard_dir()
→ load_managed_directory() // 2.15s — reads/parses all .mdb file headers
→ register_shards() // 2.0s — reads all chunk hashes, builds in-memory lookup
```
## Root cause
`ShardFileManager` is cached per `XetContext` (via `XetCommon::runtime_cache`). However, `XetSessionBuilder::build()` always constructs a brand-new `XetContext` — there is no mechanism to pass in an existing one. As a result, every `XetSession` starts with an empty shard-file-manager cache and must re-scan the full cache directory from disk.
The legacy Python-binding API (`upload_files` / `upload_bytes` in `hf_xet::legacy`) avoids this by holding a process-global `XetContext` in a `lazy_static` (`MULTITHREADED_RUNTIME`). The cache directory is scanned once per process lifetime. Any caller using `XetSessionBuilder` directly — including OpenDAL's HuggingFace service — pays the full scan cost on every upload.
## Observed via OpenDAL integration tests
Added some timings to xet-core:
```
❯ RUST_LOG=opendal_service_hf=debug,xet_data=debug OPENDAL_TEST=hf OPENDAL_HF_REPO_TYPE=bucket OPENDAL_HF_REPO_ID=kszucs/opendal RUST_TEST_THREADS=1 cargo test -p opendal --features services-hf,tests "behavior::test_write_only"
Finished `test` profile [unoptimized + debuginfo] target(s) in 0.25s
Running unittests src/lib.rs (target/debug/deps/opendal-90ceca890413e75f)
running 0 tests
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
Running tests/behavior/main.rs (target/debug/deps/behavior-338974d2e605e604)
running 1 test
2026-05-26T11:12:10.146695+02:00 DEBUG opendal_service_hf::core: core.rs:254 new_upload_commit (sync) took 30.875µs
[xet-timing] create_remote_client took 1.464291ms
[xet-timing] merge_shards_background spawn took 418.667µs
[xet-timing] load_managed_directory (710 files) took 2.038852416s
[xet-timing] shard "9b477be08c08de5844b3d67c28af0e5363c6850c6522951ce2b09ee52a3dccbb.mdb" (2976 bytes): read_all_truncated_hashes + bookkeeper update took 213.917µs
[xet-timing] shard "d8b0c4fb3f44f027bb9b6b8ad99b70a101ba80a85b4a48600ace439d214a620b.mdb" (3872 bytes): read_all_truncated_hashes + bookkeeper update took 17.125µs
[xet-timing] shard "d8eafa4b2147ed3fe3ecf10455ffd59e07783e3516ac4dd3a02d1d6dec7dc002.mdb" (4128 bytes): read_all_truncated_hashes + bookkeeper update took 14.666µs
[xet-timing] shard "940d29b15d2538298905e95ae3609545920762eb547d6015cb6c05921e94844c.mdb" (2016 bytes): read_all_truncated_hashes + bookkeeper update took 13.584µs
...
[xet-timing] register_shards (710 shards) took 1.877259125s
[xet-timing] ShardFileManager::new_in_cache_directory took 3.9170355s
[xet-timing] ShardFileManager::new_in_session_directory took 22.292µs
[xet-timing] shard_merge_jh.await took 31.25µs
[xet-timing] SessionShardInterface::new took 3.919933625s
2026-05-26T11:12:14.414164+02:00 DEBUG opendal_service_hf::core: core.rs:265 upload_commit.build() took 4.263337292s
2026-05-26T11:12:14.414193+02:00 DEBUG opendal_service_hf::writer: writer.rs:47 xet_upload_commit took 4.267583917s
2026-05-26T11:12:14.414360+02:00 DEBUG opendal_service_hf::writer: writer.rs:57 upload_stream took 157.75µs
[xet-timing] shard "c7c46df92f3e5784161b9ed2aeca2a7d3ca2427c0f247c1a9c0fdd74b0b15442.mdb" (2784 bytes): read_all_truncated_hashes + bookkeeper update took 146.25µs
[xet-timing] shard "af5685573d6e245ba3b86dcdb98e30106924fc43de2fa3b921853b06afa1e350.mdb" (2784 bytes): read_all_truncated_hashes + bookkeeper update took 156.667µs
test behavior::test_write_only ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 91 filtered out; finished in 7.14s
```
Contributor guide
Research direction
Start at XetSessionBuilder::build(), then trace XetUploadCommitBuilder::build() through FileUploadSession::new(), SessionShardInterface::new(), and ShardFileManager::new_in_cache_directory(). Compare this with the legacy hf_xet::legacy MULTITHREADED_RUNTIME path and its shared XetContext. Done means repeated upload-session builds reuse the shard cache rather than rescanning all .mdb files; verify with the OpenDAL behavior::test_write_only integration test and timing logs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100