[Feature] why partitioned mast be in table filelds?cat not cast create_time as Date
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Motivation
Paimon version
1.4.2, master
Compute Engine
Flink 1.20, paimon-kafka-connect
Minimal reproduce step:
step1: Use Debezium MySQL Connector to write customer Debezium JSON records to Kafka.
step2: then user paimon-kafka-connect to write json message to paimon.
I attempted to use create_time (formatted as yyyy-MM-dd HH:mm:ss) as a partitioning column, but that failed. Subsequently, I used PARTITIONED BY (to_date(create_time FROM 1 FOR 10)), which resulted in an error stating that the expression must be based on table columns. Is it possible to support partitioning rules like those in Doris (AUTO PARTITION BY RANGE (date_trunc(`create_time`, 'day'))) ?
Why do we use Kafka instead of Flink CDC? Because other business services need to reuse these JSON messages.
JSON Records to Kafka.eg:
{"schema":{"type":"struct","fields":[{"type":"string","optional":false,"field":"order_id"},{"type":"string","optional":true,"default":"","field":"out_order_id"},{"type":"string","optional":true,"default":"","field":"pay_order_id"},{"type":"int16","optional":true,"field":"order_status"},{"type":"string","optional":true,"default":"","field":"result_detail"},{"type":"int32","optional":false,"default":0,"field":"game_id"},{"type":"string","optional":true,"field":"game_name"},{"type":"int32","optional":true,"default":0,"field":"game_type"},{"type":"int64","optional":false,"field":"user_id"},{"type":"string","optional":true,"field":"user_account"},{"type":"int64","optional":true,"field":"account_code"},{"type":"string","optional":true,"default":"","field":"device_code"},{"type":"int16","optional":true,"default":0,"field":"platform"},{"type":"int64","optional":false,"default":0,"field":"payable_amount"},{"type":"int64","optional":false,"default":0,"field":"receipts_amount"},{"type":"int64","optional":false,"default":0,"field":"discount_amount"},{"type":"int64","optional":true,"default":0,"field":"recovery_amount"},{"type":"string","optional":true,"default":"","field":"coupon_type"},{"type":"int64","optional":true,"default":0,"field":"excise_tax"},{"type":"string","optional":true,"default":"","field":"currency"},{"type":"int64","optional":true,"field":"coupon_id"},{"type":"string","optional":true,"default":"","field":"bet_detail"},{"type":"string","optional":true,"field":"description"},{"type":"int64","optional":true,"name":"io.debezium.time.Timestamp","version":1,"field":"cancel_time"},{"type":"int64","optional":true,"name":"io.debezium.time.Timestamp","version":1,"field":"create_time"},{"type":"int64","optional":true,"name":"io.debezium.time.Timestamp","version":1,"field":"update_time"},{"type":"int64","optional":true,"field":"agent_code"},{"type":"string","optional":true,"field":"agent_account"},{"type":"string","optional":true,"field":"play_method"},{"type":"string","optional":true,"field":"number_detail"},{"type":"int64","optional":true,"field":"develop_id"},{"type":"string","optional":true,"default":"","field":"bill_date"},{"type":"int32","optional":true,"default":0,"field":"cancel_cash_times"},{"type":"string","optional":true,"default":"0","field":"issue_num"},{"type":"double","optional":true,"default":0.0,"field":"min_odds"},{"type":"string","optional":true,"default":"","field":"register_from"},{"type":"string","optional":true,"default":"","field":"login_channel"},{"type":"int64","optional":true,"default":0,"field":"main_amount"},{"type":"int64","optional":true,"default":0,"field":"redpackets_amount"},{"type":"int64","optional":true,"default":0,"field":"activity_amount"},{"type":"int64","optional":true,"default":0,"field":"activity_id"},{"type":"string","optional":true,"default":"0","field":"redpackets_detail"},{"type":"string","optional":true,"default":"0","field":"activity_type"},{"type":"int16","optional":true,"default":0,"field":"settle_status"},{"type":"int64","optional":true,"default":0,"field":"settle_amount"},{"type":"int64","optional":true,"default":0,"field":"wh_tax"},{"type":"int64","optional":true,"name":"io.debezium.time.Timestamp","version":1,"field":"settle_time"},{"type":"string","optional":false,"field":"db_name"},{"type":"string","optional":false,"field":"table_name"},{"type":"int32","optional":false,"field":"dml_type"}],"optional":false,"name":"ng_g_game_deal_order.Value"},"payload":{"order_id":"8773639782452027392102012759","out_order_id":"t8773639782411239424","pay_order_id":"1020127598773639782501179392","order_status":1,"result_detail":"","game_id":60015,"game_name":"Gold Hunter","game_type":2,"user_id":102012759,"user_account":"1730000255","account_code":10201275902416582,"device_code":"3bd7151f-0bf3-404f-bf02-119c749645ec","platform":3,"payable_amount":5000000,"receipts_amount":5000000,"discount_amount":0,"recovery_amount":0,"coupon_type":"","excise_tax":0,"currency":"NGN","coupon_id":null,"bet_detail":"","description":null,"cancel_time":null,"create_time":1786008944000,"update_time":null,"agent_code":null,"agent_account":null,"play_method":null,"number_detail":"1;1;5000000;|","develop_id":10,"bill_date":"20260806","cancel_cash_times":0,"issue_num":"0","min_odds":0.0,"register_from":"ilot","login_channel":"3","main_amount":5000000,"redpackets_amount":0,"activity_amount":0,"activity_id":0,"redpackets_detail":"0","activity_type":"0","settle_status":0,"settle_amount":0,"wh_tax":0,"settle_time":null,"db_name":"ilot_trade1","table_name":"g_game_deal_order_0","dml_type":1}}
{"schema":{"type":"struct","fields":[{"type":"string","optional":false,"field":"order_id"},{"type":"string","optional":true,"default":"","field":"out_order_id"},{"type":"string","optional":true,"default":"","field":"pay_order_id"},{"type":"int16","optional":true,"field":"order_status"},{"type":"string","optional":true,"default":"","field":"result_detail"},{"type":"int32","optional":false,"default":0,"field":"game_id"},{"type":"string","optional":true,"field":"game_name"},{"type":"int32","optional":true,"default":0,"field":"game_type"},{"type":"int64","optional":false,"field":"user_id"},{"type":"string","optional":true,"field":"user_account"},{"type":"int64","optional":true,"field":"account_code"},{"type":"string","optional":true,"default":"","field":"device_code"},{"type":"int16","optional":true,"default":0,"field":"platform"},{"type":"int64","optional":false,"default":0,"field":"payable_amount"},{"type":"int64","optional":false,"default":0,"field":"receipts_amount"},{"type":"int64","optional":false,"default":0,"field":"discount_amount"},{"type":"int64","optional":true,"default":0,"field":"recovery_amount"},{"type":"string","optional":true,"default":"","field":"coupon_type"},{"type":"int64","optional":true,"default":0,"field":"excise_tax"},{"type":"string","optional":true,"default":"","field":"currency"},{"type":"int64","optional":true,"field":"coupon_id"},{"type":"string","optional":true,"default":"","field":"bet_detail"},{"type":"string","optional":true,"field":"description"},{"type":"int64","optional":true,"name":"io.debezium.time.Timestamp","version":1,"field":"cancel_time"},{"type":"int64","optional":true,"name":"io.debezium.time.Timestamp","version":1,"field":"create_time"},{"type":"int64","optional":true,"name":"io.debezium.time.Timestamp","version":1,"field":"update_time"},{"type":"int64","optional":true,"field":"agent_code"},{"type":"string","optional":true,"field":"agent_account"},{"type":"string","optional":true,"field":"play_method"},{"type":"string","optional":true,"field":"number_detail"},{"type":"int64","optional":true,"field":"develop_id"},{"type":"string","optional":true,"default":"","field":"bill_date"},{"type":"int32","optional":true,"default":0,"field":"cancel_cash_times"},{"type":"string","optional":true,"default":"0","field":"issue_num"},{"type":"double","optional":true,"default":0.0,"field":"min_odds"},{"type":"string","optional":true,"default":"","field":"register_from"},{"type":"string","optional":true,"default":"","field":"login_channel"},{"type":"int64","optional":true,"default":0,"field":"main_amount"},{"type":"int64","optional":true,"default":0,"field":"redpackets_amount"},{"type":"int64","optional":true,"default":0,"field":"activity_amount"},{"type":"int64","optional":true,"default":0,"field":"activity_id"},{"type":"string","optional":true,"default":"0","field":"redpackets_detail"},{"type":"string","optional":true,"default":"0","field":"activity_type"},{"type":"int16","optional":true,"default":0,"field":"settle_status"},{"type":"int64","optional":true,"default":0,"field":"settle_amount"},{"type":"int64","optional":true,"default":0,"field":"wh_tax"},{"type":"int64","optional":true,"name":"io.debezium.time.Timestamp","version":1,"field":"settle_time"},{"type":"string","optional":false,"field":"db_name"},{"type":"string","optional":false,"field":"table_name"},{"type":"int32","optional":false,"field":"dml_type"}],"optional":false,"name":"ng_g_game_deal_order.Value"},"payload":{"order_id":"8773639782452027392102012759","out_order_id":"t8773639782411239424","pay_order_id":"1020127598773639782501179392","order_status":1,"result_detail":"","game_id":60015,"game_name":"Gold Hunter","game_type":2,"user_id":102012759,"user_account":"1730000255","account_code":10201275902416582,"device_code":"3bd7151f-0bf3-404f-bf02-119c749645ec","platform":3,"payable_amount":5000000,"receipts_amount":5000000,"discount_amount":0,"recovery_amount":0,"coupon_type":"","excise_tax":0,"currency":"NGN","coupon_id":null,"bet_detail":"","description":null,"cancel_time":null,"create_time":1786008944000,"update_time":1786008956000,"agent_code":null,"agent_account":null,"play_method":null,"number_detail":"1;1;5000000;|","develop_id":10,"bill_date":"20260806","cancel_cash_times":0,"issue_num":"0","min_odds":0.0,"register_from":"ilot","login_channel":"3","main_amount":5000000,"redpackets_amount":0,"activity_amount":0,"activity_id":0,"redpackets_detail":"0","activity_type":"0","settle_status":3,"settle_amount":7500000,"wh_tax":0,"settle_time":1786008956000,"db_name":"ilot_trade1","table_name":"g_game_deal_order_0","dml_type":2}}
### Solution
_No response_
### Anything else?
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the partitioning attempts on Paimon 1.4.2 or master with Flink 1.20 and paimon-kafka-connect, using create_time from the Debezium JSON records. Trace the validation of partition expressions and schema handling. Done means the requested date-based partitioning behavior is supported and verified with the provided create_time examples.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kafka, sql
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100