apache / apache/paimon

[Feature] defined mongo cdc schema on table_conf

Open
#4,695 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Motivation

according to this issue https://github.com/apache/paimon/issues/4683

we find that mongo cdc change schema frequently,and can not query right value on starrock paimon cagalog

### Solution
we can defined a schema by set permanent type and filed to save it

like seatunnel we can defined a schema on mongo cdc table_conf
```

/bin/flink run \
/path/to/paimon-flink-action-0.9.0.jar \
mongodb_sync_table \
--warehouse hdfs:///path/to/warehouse \
--database test_db \
--table test_table \
--partition_keys pt \
--computed_column '_year=year(age)' \
--mongodb_conf hosts=127.0.0.1:27017 \
--mongodb_conf username=root \
--mongodb_conf password=123456 \
--mongodb_conf database=source_db \
--mongodb_conf collection=source_table1 \
--catalog_conf metastore=hive \
--catalog_conf uri=thrift://hive-metastore:9083 \
--table_conf bucket=4 \
--table_conf changelog-producer=input \
--table_conf sink.parallelism=4 \
--table_conf schema={
fields {
"_id" : int,
"name" : string,
"created" : datetime,
"isDelete" : boolean
}
}
```

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing issue 4683 and the MongoDB sync table command shown in this issue. Determine how a permanent schema supplied through table_conf should be represented and applied; done means MongoDB CDC schema changes no longer cause incorrect query values in the Starrocks Paimon catalog.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, mongodb
Domain
data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.