[Consult] [Lance] Consult for implementation and benefit of pamion integrate with lance file format

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Question

Hi @leaves12138, as for integrating paimon with lance format(here i see consider lance file format firstly), is there some detail design doc as for this implementation.

**I have some questions as for implementation and benefit like follows**:
> Here i see in paimon repo, there are mainly these two pull request:
> https://github.com/apache/paimon/pull/5760/files
> https://github.com/apache/paimon/pull/5809/files

- As for above Paimon integration implementation, is that just a part of implementation? as here i see it seems not an end-to-end implementation(seems haven't add spark/flink test case to verify).

- **What's kinds of performance benefit may obtain from above implementation**(using lance file format instead of parquet), and why we can obtain these benefit. Here i thought the good performance of random access in lance may because of serveral part of index ( table level index、file level index(chunk index) and fragment level index(i am not sure whether exist)), here in current integration, directly integrate with lance file, may be can only benefit from file level chunk index ? is there other benifit we can obtain from integrate with lance file ?

- Here i see the implementation mainly implemented generic file reader API, **may be this can not support random access well in lance file, do you think an much more lance native reader is needed**, do you think this required a lot of work and effort.

- Here paimon mainly implement with Java, and Lance with Rust, here i see lance repo provide jni implementation as following, and i want to know how much will it effet with java -call> rust instead of native rust call in lance, do you have a approximate or tested performance result on random access or full scan.
- https://github.com/lancedb/lance/tree/main/java/core/lance-jni

- As for currently implementation, does it can only integrate paimon with lance Data File, but can not integrate delete file in the same time, thus unable support hight performance delete ? and unable to support hight performance column update(as all column in one lance data file) ?

Start by reading pull requests 5760 and 5809, then inspect Lance's Java JNI implementation linked in the issue and the existing Paimon integration. The issue asks about end-to-end Spark and Flink coverage, performance, random access, deletes, and updates, but gives no concrete change or acceptance criteria, so completion is not defined.

Written by the indexing model from the issue text.