[FEATURE] Add a generic read-only JDBC catalog based on standard JDBC metadata
- Dominant language
- Java
- Stars
- 3.2k
- Forks
- 935
- Avg merge
- 1d 17h
- Merged PRs (30d)
- 339
Description
## Motivation
Currently, Gravitino provides dedicated JDBC catalogs for some relational databases, such as MySQL and PostgreSQL.
However, many users also need to access metadata from other JDBC-compatible databases, such as Oracle, SQL Server, DB2, Kingbase, DM, GaussDB, and others.
Developing a dedicated catalog for every JDBC-compatible database has a high cost. In many cases, users only need read-only metadata discovery, such as schemas, tables, and columns.
So it would be useful to provide a generic read-only JDBC catalog.
## Proposal
Add a generic JDBC catalog provider.
The catalog should read metadata through standard JDBC APIs, mainly `java.sql.DatabaseMetaData`.
The first version should be read-only.
Supported capabilities:
- List schemas
- List tables
- Load table metadata
- Load column metadata
- Map standard JDBC types to Gravitino types
- Preserve unknown or vendor-specific types as external types
Example JDBC metadata APIs:
```java
DatabaseMetaData metadata = connection.getMetaData();
metadata.getSchemas();
metadata.getTables(catalog, schemaPattern, tableNamePattern, new String[] {"TABLE", "VIEW"});
metadata.getColumns(catalog, schemaPattern, tableNamePattern, columnNamePattern);
metadata.getPrimaryKeys(catalog, schema, table);
```
## Type Mapping
The generic JDBC catalog should mainly use standard `java.sql.Types` for type mapping.
Example mapping:
| JDBC Type | Gravitino Type |
| --- | --- |
| `BOOLEAN`, `BIT` | Boolean |
| `TINYINT` | Byte |
| `SMALLINT` | Short |
| `INTEGER` | Integer |
| `BIGINT` | Long |
| `FLOAT`, `REAL` | Float |
| `DOUBLE` | Double |
| `DECIMAL`, `NUMERIC` | Decimal |
| `CHAR`, `VARCHAR`, `NCHAR`, `NVARCHAR` | String / Varchar |
| `DATE` | Date |
| `TIME` | Time |
| `TIMESTAMP` | Timestamp |
| `BINARY`, `VARBINARY`, `BLOB` | Binary |
| Unknown or vendor-specific types | External type |
The mapping should be conservative. If a JDBC type cannot be safely mapped, it should be kept as an external type instead of being incorrectly converted.
## Non-goals
The first version should not support write operations.
Non-goals:
- Create schema
- Drop schema
- Create table
- Drop table
- Alter table
- Update table properties
- Full dialect-specific type mapping
- Vendor-specific DDL parsing
- Partition metadata
- Statistics metadata
- Stored procedure metadata
This generic JDBC catalog is not intended to replace existing dedicated JDBC catalogs, such as MySQL or PostgreSQL.
Dedicated catalogs should still be used when full dialect-specific support is required.
## Benefits
This feature can provide a lightweight way to discover metadata from JDBC-compatible databases that do not yet have dedicated Gravitino catalogs.
It can reduce the initial cost of integrating long-tail relational databases into Gravitino.
It also allows users to use Gravitino for basic metadata browsing and downstream metadata usage without developing a new catalog for every JDBC database.
## Example Configuration
```properties
provider = jdbc-generic
jdbc-url = jdbc:xxx://host:port/database
jdbc-driver = com.example.Driver
jdbc-user = user
jdbc-password = password
jdbc.metadata.catalog-pattern = *
jdbc.metadata.schema-pattern = *
jdbc.metadata.table-types = TABLE,VIEW
```
Optional schema filters:
```properties
jdbc.metadata.include-schemas = schema1,schema2
jdbc.metadata.exclude-schemas = SYS,SYSTEM,INFORMATION_SCHEMA
```
## Summary
Add a generic read-only JDBC catalog based on standard JDBC metadata APIs.
The goal is to provide best-effort metadata discovery for JDBC-compatible databases that do not yet have dedicated Gravitino catalogs.
Contributor guide
Research direction
Start by reviewing the existing MySQL and PostgreSQL catalogs, then trace the standard java.sql.DatabaseMetaData APIs listed in the proposal. Done means a generic read-only JDBC catalog can list schemas and tables, load table and column metadata, map standard JDBC types conservatively, and preserve unknown types as external types.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100