Support HDFS as a storage backend

Background

HDFS is widely used in enterprise data platforms, and many teams want to use Lance as their columnar format on top of HDFS. Full HDFS support is currently blocked by two upstream dependencies.

Related PR

• lance-format/lance#5472 — HDFS support for Lance

Blocking dependencies

This PR is currently blocked by two upstream issues:

1. hdrs — missing HDFS copy support

The hdrs crate (used by Lance's HDFS object store provider) does not implement the copy operation. This causes failures when Lance needs to copy objects on HDFS, for example during optimize_indice on inverted indexes.

• Upstream PR: Xuanwo/hdrs#143

2. opendal — rename_with_if_not_exist

HdfsSafeCommitHandler need this.

• Upstream PR: apache/opendal#7815

Call for help

We'd appreciate the community's help in reviewing and merging the two upstream PRs so that HDFS support can land in Lance. If there are alternative approaches or workarounds, please share them in this issue or the PR discussion.

Start by reading the related Lance PR #5472 and the blocking hdrs#143 and opendal#7815 discussions. Check whether hdrs copy support and opendal's rename_with_if_not_exist support have landed, then verify the HDFS path and optimize_indice scenario described here. Done means the upstream blockers are resolved and HDFS support can land in Lance.