[Feature] defined mongo cdc schema on table_conf
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Motivation
according to this issue https://github.com/apache/paimon/issues/4683
we find that mongo cdc change schema frequently,and can not query right value on starrock paimon cagalog
### Solution
we can defined a schema by set permanent type and filed to save it
like seatunnel we can defined a schema on mongo cdc table_conf
```
/bin/flink run \
/path/to/paimon-flink-action-0.9.0.jar \
mongodb_sync_table \
--warehouse hdfs:///path/to/warehouse \
--database test_db \
--table test_table \
--partition_keys pt \
--computed_column '_year=year(age)' \
--mongodb_conf hosts=127.0.0.1:27017 \
--mongodb_conf username=root \
--mongodb_conf password=123456 \
--mongodb_conf database=source_db \
--mongodb_conf collection=source_table1 \
--catalog_conf metastore=hive \
--catalog_conf uri=thrift://hive-metastore:9083 \
--table_conf bucket=4 \
--table_conf changelog-producer=input \
--table_conf sink.parallelism=4 \
--table_conf schema={
fields {
"_id" : int,
"name" : string,
"created" : datetime,
"isDelete" : boolean
}
}
```
### Anything else?
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reviewing issue 4683 and the MongoDB sync table command shown in this issue. Determine how a permanent schema supplied through table_conf should be represented and applied; done means MongoDB CDC schema changes no longer cause incorrect query values in the Starrocks Paimon catalog.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, mongodb
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100