Slow dataset_load (~4.9s) on HDFS-backed indexed dataset with 2000 fragments (lance_ray)

Summary

During lance_ray vector search, the dataset_load worker phase takes ~4.86 seconds when opening an indexed Lance dataset on HDFS with 2000 fragments and dataset version 1007. This cost affects worker startup latency for search workloads.

We would like to know whether this duration is expected at this scale, and whether there are recommended optimizations (metadata caching, fewer HDFS round-trips, manifest handling, etc.).

Environment

• Storage: HDFS via OpenDAL (opendal::services::hdfs, hdrs)
• NameNode: hdfs://bzl-hdfs
• Dataset URI: hdfs://bzl-hdfs/user/platform_rd_two/zhb/bulk_50b_lance_embedding.lance
• Component: lance_ray.search vector search worker (phase=dataset_load)
• Date: 2026-06-01

Dataset characteristics

Observed behavior

1. HDFS NameNode connection takes ~2s at the start of the phase.
2. The same manifest path is opened and closed three times before load completes.
3. Load eventually succeeds (lance::dataset_events, status="success").

Manifest path uses u64::MAX - version

Debug logs repeatedly open:

/user/platform_rd_two/zhb/bulk_50b_lance_embedding.lance/_versions/18446744073709550608.manifest

Note: 18446744073709550608 equals u64::MAX - 1007, while the loaded version in events is 1007. Is this sentinel/offset path expected during version resolution? Could redundant HDFS probes add latency?

Logs

INFO:lance_ray.search:Vector search worker phase=dataset_load start: kind=indexed, fragments=2000, index_segments=1, dataset_version=1007
[2026-06-01T07:08:49Z DEBUG hdrs::client] connect name node hdfs://bzl-hdfs
[2026-06-01T07:08:51Z DEBUG hdrs::client] name node hdfs://bzl-hdfs connected
[2026-06-01T07:08:51Z DEBUG opendal::services::hdfs::backend] backend build started: HdfsBuilder { config: HdfsConfig { root: Some("/"), name_node: Some("hdfs://bzl-hdfs"), kerberos_ticket_cache_path: None, user: Some("platform_rd_two"), enable_append: false, atomic_write_dir: None, .. } }
[2026-06-01T07:08:51Z DEBUG opendal::services::hdfs::backend] backend use root /
[2026-06-01T07:08:51Z DEBUG hdrs::client] connect name node hdfs://bzl-hdfs
[2026-06-01T07:08:51Z DEBUG hdrs::client] name node hdfs://bzl-hdfs connected
[2026-06-01T07:08:51Z DEBUG hdrs::open_options] open file /user/platform_rd_two/zhb/bulk_50b_lance_embedding.lance/_versions/18446744073709550608.manifest with flags 524288
[2026-06-01T07:08:51Z DEBUG hdrs::open_options] file /user/platform_rd_two/zhb/bulk_50b_lance_embedding.lance/_versions/18446744073709550608.manifest with flags 524288 opened
[2026-06-01T07:08:51Z DEBUG hdrs::file] file has been closed
[2026-06-01T07:08:51Z DEBUG hdrs::open_options] open file /user/platform_rd_two/zhb/bulk_50b_lance_embedding.lance/_versions/18446744073709550608.manifest with flags 524288
[2026-06-01T07:08:51Z DEBUG hdrs::open_options] file /user/platform_rd_two/zhb/bulk_50b_lance_embedding.lance/_versions/18446744073709550608.manifest with flags 524288 opened
[2026-06-01T07:08:51Z DEBUG hdrs::file] file has been closed
[2026-06-01T07:08:51Z DEBUG hdrs::open_options] open file /user/platform_rd_two/zhb/bulk_50b_lance_embedding.lance/_versions/18446744073709550608.manifest with flags 524288
[2026-06-01T07:08:51Z DEBUG hdrs::open_options] file /user/platform_rd_two/zhb/bulk_50b_lance_embedding.lance/_versions/18446744073709550608.manifest with flags 524288 opened
[2026-06-01T07:08:53Z DEBUG hdrs::file] file has been closed
[2026-06-01T07:08:54Z INFO  lance::events] target="lance::dataset_events" event="loading" uri="hdfs://bzl-hdfs/user/platform_rd_two/zhb/bulk_50b_lance_embedding.lance" target_ref=Some(VersionNumber(1007)) version=1007 status="success"
INFO:lance_ray.search:Vector search worker phase=dataset_load done: duration_ms=4859.355, kind=indexed, fragments=2000, index_segments=1

Questions

1. Is ~5s dataset_load expected for an indexed dataset with 2000 fragments on HDFS?
2. Can manifest/metadata reads be cached or batched across workers to avoid repeated HDFS opens?
3. Is the _versions/18446744073709550608.manifest path intentional, and could failed or duplicate probes add latency?

Impact

Each vector search worker pays this cost at dataset_load, affecting latency and throughput when workers are spawned frequently.

Begin with the lance_ray.search dataset_load entry point and the dataset version-resolution and manifest reads shown in the logs; compare the three HDFS opens with the NameNode connection cost. Use the reported 2000-fragment, version-1007 workload to determine whether the sentinel manifest path and repeated reads are expected, and document or validate a measurable reduction in worker startup latency.