[FEATURE] Provide Batch Load Api for Multiple Entities
- Dominant language
- Java
- Stars
- 3.2k
- Forks
- 935
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 298
Description
### Describe the feature
At present, Gravitino only provides APIs for precisely querying the details of an entity based on a specific NameIdentifier (such as getTable, getPartition, loadFileset, etc.), but does not offer an API for batch querying a subset of entities within a schema based on a List.
For users, batch querying is a very common requirement. Currently, for example, if a user wants to retrieve the details of n tables, they would need to call the getTable API n times, which is unnecessary and factors such as network jitter may cause some responses to slow down. If we implement a batch query interface, it would greatly reduce the number of RPC calls and also significantly decrease the time spent in RPC network traffic.
For metadata management systems that we are familiar with, such as HiveMetaStore which Gravitino is using, they have implemented batch query interfaces, such as:
```java
List getTableObjectsByName(String dbName, List tableNames);
List getPartitionsByNames(String db_name, String tbl_name, List part_names);
```
Therefore, we believe that for Gravitino, as a high-performance and federated metadata lake, it is necessary to implement a batch query interface for various entities (table, partition, fileset, topic, etc.) within a schema.
Currently, due to the batch query requirements in actual production applications, we have preliminarily implemented a batch query interface for filesets
```java
Fileset[] loadFilesetList(NameIdentifier[] idents);
```
I have already submitted a PR, and I wish to contribute it to the community if the community is interested. We can discuss together whether this is a good implementation and whether it can be applied to other data sources.
### Motivation
_No response_
### Describe the solution
_No response_
### Additional context
_No response_
Contributor guide
Research direction
Review the existing getTable, getPartition, and loadFileset entry points, along with the proposed loadFilesetList(NameIdentifier[]) interface and the submitted fileset implementation. Then determine the scope for batch querying tables, partitions, filesets, and topics. Done requires an agreed API design and validated behavior for the supported entities.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- api, backend-api-design
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100