Create Persisted Generic Catalog.
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 522
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 137
Description
### Is your feature request related to a problem? Please describe.
i'm asking pretty please that a catalog type=generic be added.
That allows for the definition of **Kafka**, **PostgreSQL**, **MySQL**, **MongoDB**, etc. transcient sources thats backed by **Flink CDC** from said source.
The defined object wont store anything, per say. the **type=generic** will be like the current **type=generic_in_memory** catalog, just permanently persisted.
basically the below...
```
CREATE CATALOG c_cdcsource WITH
('type'='generic');
```
which allows all objects defined like the following, have their definition span sessions.
```
USE CATALOG c_cdcsource;
CREATE DATABASE IF NOT EXISTS demog;
USE demog;
CREATE OR REPLACE TABLE accountholders (
_id BIGINT NOT NULL
,nationalid VARCHAR(16) NOT NULL
,firstname VARCHAR(100)
,lastname VARCHAR(100)
,dob VARCHAR(10)
,gender VARCHAR(10)
,children INT
,address STRING
,accounts STRING
,emailaddress VARCHAR(100)
,mobilephonenumber VARCHAR(20)
,created_at TIMESTAMP_LTZ(3)
,WATERMARK FOR created_at AS created_at - INTERVAL '15' SECOND
,PRIMARY KEY (_id) NOT ENFORCED
) WITH (
'connector' = 'postgres-cdc'
,'hostname' = 'postgrescdc'
,'port' = '5432'
,'username' = 'dbadmin'
,'password' = 'dbpassword'
,'database-name' = 'demog'
,'schema-name' = 'public'
,'table-name' = 'accountholders'
,'slot.name' = 'accountholders_pyflink' -- Can't include capital letters
,'scan.incremental.snapshot.enabled' = 'true'
,'scan.startup.mode' = 'initial'
,'decoding.plugin.name' = 'pgoutput'
);
```
### Describe the solution you'd like
_No response_
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
Contributor guide
Research direction
Start by locating the existing type=generic_in_memory catalog implementation and its persistence/session lifecycle; no source files or tests are named in the issue. Done means a type=generic catalog can define the shown objects and retain their definitions across sessions for the listed CDC-backed sources.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kafka, mongodb, mysql, postgresql, sql
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100