apache / apache/polaris

Create Persisted Generic Catalog.

Open
#3,342 3 comments 0 reactions 0 assignees View on GitHub
enhancement stale
Dominant language
Java
Stars
2.1k
Forks
522
Avg merge
1d 22h
Merged PRs (30d)
137

Description

### Is your feature request related to a problem? Please describe.

i'm asking pretty please that a catalog type=generic be added.

That allows for the definition of **Kafka**, **PostgreSQL**, **MySQL**, **MongoDB**, etc. transcient sources thats backed by **Flink CDC** from said source.

The defined object wont store anything, per say. the **type=generic** will be like the current **type=generic_in_memory** catalog, just permanently persisted.

basically the below...

```
CREATE CATALOG c_cdcsource WITH
('type'='generic');
```
which allows all objects defined like the following, have their definition span sessions.

```
USE CATALOG c_cdcsource;
CREATE DATABASE IF NOT EXISTS demog;
USE demog;

CREATE OR REPLACE TABLE accountholders (
_id BIGINT NOT NULL
,nationalid VARCHAR(16) NOT NULL
,firstname VARCHAR(100)
,lastname VARCHAR(100)
,dob VARCHAR(10)
,gender VARCHAR(10)
,children INT
,address STRING
,accounts STRING
,emailaddress VARCHAR(100)
,mobilephonenumber VARCHAR(20)
,created_at TIMESTAMP_LTZ(3)
,WATERMARK FOR created_at AS created_at - INTERVAL '15' SECOND
,PRIMARY KEY (_id) NOT ENFORCED
) WITH (
'connector' = 'postgres-cdc'
,'hostname' = 'postgrescdc'
,'port' = '5432'
,'username' = 'dbadmin'
,'password' = 'dbpassword'
,'database-name' = 'demog'
,'schema-name' = 'public'
,'table-name' = 'accountholders'
,'slot.name' = 'accountholders_pyflink' -- Can't include capital letters
,'scan.incremental.snapshot.enabled' = 'true'
,'scan.startup.mode' = 'initial'
,'decoding.plugin.name' = 'pgoutput'
);
```

### Describe the solution you'd like

_No response_

### Describe alternatives you've considered

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by locating the existing type=generic_in_memory catalog implementation and its persistence/session lifecycle; no source files or tests are named in the issue. Done means a type=generic catalog can define the shown objects and retain their definitions across sessions for the listed CDC-backed sources.

Written by the indexing model from the issue text.

Assessment

Tech stack
kafka, mongodb, mysql, postgresql, sql
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.