apache / apache/gravitino

[Improvement] Support configuring multiple HMS URIs for different network scenarios

Open
#7,953 4 comments 0 reactions 1 assignee Claimed by @FANNG1 View on GitHub
feature
Dominant language
Java
Stars
3.2k
Forks
935
Avg merge
1d 17h
Merged PRs (30d)
286

Description

### Describe the feature

In cloud environments, it is common to have Gravitino server, Hive Metastore (HMS), and Spark engines deployed in different VPCs. In my scenario, the Gravitino server and HMS are in the same VPC and communicate via the internal network, while Spark engines are in another VPC and access HMS through a load balancer (LB). Currently, Gravitino only allows configuring a single HMS URI, which makes it difficult to support both internal and external access patterns at the same time.
I would like to request support for configuring multiple HMS URIs, so that Gravitino can use the internal URI to communicate with HMS, while exposing the external URI (e.g., via LB) to compute engines like Spark. This would greatly improve flexibility and compatibility in multi-VPC or hybrid cloud deployments.

### Motivation

Supporting multiple HMS URIs is important for real-world cloud deployments where network isolation and security are required. In many cases, the Gravitino server and HMS need to communicate over a private network for security and performance, while compute engines (such as Spark, Flink, Trino, etc.) may only be able to access HMS via a public endpoint or load balancer. Without the ability to configure both internal and external HMS URIs, users are forced to make trade-offs that can compromise either security or accessibility

### Describe the solution

Allow users to configure both an internal HMS URI (for Gravitino server to HMS communication) and an external HMS URI (for compute engines to access HMS via Gravitino). Gravitino should use the internal URI for its own operations, but when returning HMS-related metadata or endpoints to clients (such as Spark), it should provide the external URI. This could be implemented by adding separate configuration options for internal and external HMS URIs, and updating the relevant logic to use the appropriate URI based on the context.

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.