apache / apache/gravitino

[FEATURE] Provide Batch Load Api for Multiple Entities

Open
#3,989 4 comments 1 reaction 0 assignees View on GitHub
feature
Dominant language
Java
Stars
3.2k
Forks
935
Avg merge
1d 16h
Merged PRs (30d)
298

Description

### Describe the feature

At present, Gravitino only provides APIs for precisely querying the details of an entity based on a specific NameIdentifier (such as getTable, getPartition, loadFileset, etc.), but does not offer an API for batch querying a subset of entities within a schema based on a List.

For users, batch querying is a very common requirement. Currently, for example, if a user wants to retrieve the details of n tables, they would need to call the getTable API n times, which is unnecessary and factors such as network jitter may cause some responses to slow down. If we implement a batch query interface, it would greatly reduce the number of RPC calls and also significantly decrease the time spent in RPC network traffic.

For metadata management systems that we are familiar with, such as HiveMetaStore which Gravitino is using, they have implemented batch query interfaces, such as:

```java
List getTableObjectsByName(String dbName, List tableNames);
List getPartitionsByNames(String db_name, String tbl_name, List part_names);
```

Therefore, we believe that for Gravitino, as a high-performance and federated metadata lake, it is necessary to implement a batch query interface for various entities (table, partition, fileset, topic, etc.) within a schema.

Currently, due to the batch query requirements in actual production applications, we have preliminarily implemented a batch query interface for filesets

```java
Fileset[] loadFilesetList(NameIdentifier[] idents);
```

I have already submitted a PR, and I wish to contribute it to the community if the community is interested. We can discuss together whether this is a good implementation and whether it can be applied to other data sources.

### Motivation

_No response_

### Describe the solution

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Review the existing getTable, getPartition, and loadFileset entry points, along with the proposed loadFilesetList(NameIdentifier[]) interface and the submitted fileset implementation. Then determine the scope for batch querying tables, partitions, filesets, and topics. Done requires an agreed API design and validated behavior for the supported entities.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
api, backend-api-design
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.