apache / apache/gravitino

[FEATURE] Add a generic read-only JDBC catalog based on standard JDBC metadata

Open
#11,453 1 comment 0 reactions 0 assignees View on GitHub
feature
Dominant language
Java
Stars
3.2k
Forks
935
Avg merge
1d 17h
Merged PRs (30d)
339

Description

## Motivation

Currently, Gravitino provides dedicated JDBC catalogs for some relational databases, such as MySQL and PostgreSQL.

However, many users also need to access metadata from other JDBC-compatible databases, such as Oracle, SQL Server, DB2, Kingbase, DM, GaussDB, and others.

Developing a dedicated catalog for every JDBC-compatible database has a high cost. In many cases, users only need read-only metadata discovery, such as schemas, tables, and columns.

So it would be useful to provide a generic read-only JDBC catalog.

## Proposal

Add a generic JDBC catalog provider.

The catalog should read metadata through standard JDBC APIs, mainly `java.sql.DatabaseMetaData`.

The first version should be read-only.

Supported capabilities:

- List schemas
- List tables
- Load table metadata
- Load column metadata
- Map standard JDBC types to Gravitino types
- Preserve unknown or vendor-specific types as external types

Example JDBC metadata APIs:

```java
DatabaseMetaData metadata = connection.getMetaData();

metadata.getSchemas();
metadata.getTables(catalog, schemaPattern, tableNamePattern, new String[] {"TABLE", "VIEW"});
metadata.getColumns(catalog, schemaPattern, tableNamePattern, columnNamePattern);
metadata.getPrimaryKeys(catalog, schema, table);
```

## Type Mapping

The generic JDBC catalog should mainly use standard `java.sql.Types` for type mapping.

Example mapping:

| JDBC Type | Gravitino Type |
| --- | --- |
| `BOOLEAN`, `BIT` | Boolean |
| `TINYINT` | Byte |
| `SMALLINT` | Short |
| `INTEGER` | Integer |
| `BIGINT` | Long |
| `FLOAT`, `REAL` | Float |
| `DOUBLE` | Double |
| `DECIMAL`, `NUMERIC` | Decimal |
| `CHAR`, `VARCHAR`, `NCHAR`, `NVARCHAR` | String / Varchar |
| `DATE` | Date |
| `TIME` | Time |
| `TIMESTAMP` | Timestamp |
| `BINARY`, `VARBINARY`, `BLOB` | Binary |
| Unknown or vendor-specific types | External type |

The mapping should be conservative. If a JDBC type cannot be safely mapped, it should be kept as an external type instead of being incorrectly converted.

## Non-goals

The first version should not support write operations.

Non-goals:

- Create schema
- Drop schema
- Create table
- Drop table
- Alter table
- Update table properties
- Full dialect-specific type mapping
- Vendor-specific DDL parsing
- Partition metadata
- Statistics metadata
- Stored procedure metadata

This generic JDBC catalog is not intended to replace existing dedicated JDBC catalogs, such as MySQL or PostgreSQL.

Dedicated catalogs should still be used when full dialect-specific support is required.

## Benefits

This feature can provide a lightweight way to discover metadata from JDBC-compatible databases that do not yet have dedicated Gravitino catalogs.

It can reduce the initial cost of integrating long-tail relational databases into Gravitino.

It also allows users to use Gravitino for basic metadata browsing and downstream metadata usage without developing a new catalog for every JDBC database.

## Example Configuration

```properties
provider = jdbc-generic

jdbc-url = jdbc:xxx://host:port/database
jdbc-driver = com.example.Driver
jdbc-user = user
jdbc-password = password

jdbc.metadata.catalog-pattern = *
jdbc.metadata.schema-pattern = *
jdbc.metadata.table-types = TABLE,VIEW
```

Optional schema filters:

```properties
jdbc.metadata.include-schemas = schema1,schema2
jdbc.metadata.exclude-schemas = SYS,SYSTEM,INFORMATION_SCHEMA
```

## Summary

Add a generic read-only JDBC catalog based on standard JDBC metadata APIs.

The goal is to provide best-effort metadata discovery for JDBC-compatible databases that do not yet have dedicated Gravitino catalogs.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the existing MySQL and PostgreSQL catalogs, then trace the standard java.sql.DatabaseMetaData APIs listed in the proposal. Done means a generic read-only JDBC catalog can list schemas and tables, load table and column metadata, map standard JDBC types conservatively, and preserve unknown types as external types.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.