Should we provide a query interface from rowId to rowAddr to the non-Rust side?

The stable rowId is a crucial feature of Lance, giving it usable primary key functionality. Currently, the rowIdIndex is a completely internal data structure that plays a vital role in various internal operations such as update, take, and indexed queries. However, these operations are at the Dataset level and cannot be directly utilized by big data engines like Flink, Spark, or Ray.

For example, in the column addition scenario, Lance needs to align new data with existing rows based on rowId, as illustrated below:

In distributed computing, we need to horizontally partition the dataset for computation by rows (i.e., Fragments), which requires shuffling the data.

Therefore, we need the ability to convert rowId to rowAddress at the computing engine level, so that we can further obtain the FragmentId. Currently, there is no ready-to-use interface for this purpose. As a workaround, I have to SELECT all <RowId, RowAddress> pairs from the source table and perform a join with the new data to obtain the corresponding RowAddresses. However, this join incurs significant overhead, and even if the column addition involves only a small subset of rows, it still requires scanning the entire source table, which is highly inefficient.

Therefore, I'm wondering if we could expose an interface in the Dataset to query the mapping from rowId to rowAddr:

impl Dataset {
  pub async fn getRowAddress(&self, row_id: u64) -> Option<RowAddress> {
      ...
  }
  // get batch to reduce jni cost
  pub async fn getBatchRowAddress(&self, row_ids: &[u64]) -> &[RowAddress] {
      ...
  }
}

Or alternatively, implement a delegated row ID index at the Java/Python FFI (Rust) boundary to enable efficient rowId-to-rowAddr lookups from non-Rust environments.

Struct LanceRowIdIndex {
    inner: RowIdIndex
}

impl LanceRowIdIndex {
    pub async fn open(dataset: &Dataset) -> LanceRowIdIndex {
        ...
    }
  
    pub async fn get(&self, row_id: u64) -> Option<RowAddress> {
        ...
    }

    // get batch to reduce jni cost
    pub async fn getBatch(&self, row_ids: &[u64]) -> &[RowAddress] {
         ...
    }
}

Do you think this functionality is necessary? @majin1102 @jackye1995 @yanghua

I'd be very happy to implement this feature!