apache / apache/paimon

[Feature] why partitioned mast be in table filelds?cat not cast create_time as Date

Open
#9,070 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Motivation

Paimon version
1.4.2, master

Compute Engine
Flink 1.20, paimon-kafka-connect

Minimal reproduce step:
step1: Use Debezium MySQL Connector to write customer Debezium JSON records to Kafka.

step2: then user paimon-kafka-connect to write json message to paimon.

I attempted to use create_time (formatted as yyyy-MM-dd HH:mm:ss) as a partitioning column, but that failed. Subsequently, I used PARTITIONED BY (to_date(create_time FROM 1 FOR 10)), which resulted in an error stating that the expression must be based on table columns. Is it possible to support partitioning rules like those in Doris (AUTO PARTITION BY RANGE (date_trunc(`create_time`, 'day'))) ?

Why do we use Kafka instead of Flink CDC? Because other business services need to reuse these JSON messages.

JSON Records to Kafka.eg:

{"schema":{"type":"struct","fields":[{"type":"string","optional":false,"field":"order_id"},{"type":"string","optional":true,"default":"","field":"out_order_id"},{"type":"string","optional":true,"default":"","field":"pay_order_id"},{"type":"int16","optional":true,"field":"order_status"},{"type":"string","optional":true,"default":"","field":"result_detail"},{"type":"int32","optional":false,"default":0,"field":"game_id"},{"type":"string","optional":true,"field":"game_name"},{"type":"int32","optional":true,"default":0,"field":"game_type"},{"type":"int64","optional":false,"field":"user_id"},{"type":"string","optional":true,"field":"user_account"},{"type":"int64","optional":true,"field":"account_code"},{"type":"string","optional":true,"default":"","field":"device_code"},{"type":"int16","optional":true,"default":0,"field":"platform"},{"type":"int64","optional":false,"default":0,"field":"payable_amount"},{"type":"int64","optional":false,"default":0,"field":"receipts_amount"},{"type":"int64","optional":false,"default":0,"field":"discount_amount"},{"type":"int64","optional":true,"default":0,"field":"recovery_amount"},{"type":"string","optional":true,"default":"","field":"coupon_type"},{"type":"int64","optional":true,"default":0,"field":"excise_tax"},{"type":"string","optional":true,"default":"","field":"currency"},{"type":"int64","optional":true,"field":"coupon_id"},{"type":"string","optional":true,"default":"","field":"bet_detail"},{"type":"string","optional":true,"field":"description"},{"type":"int64","optional":true,"name":"io.debezium.time.Timestamp","version":1,"field":"cancel_time"},{"type":"int64","optional":true,"name":"io.debezium.time.Timestamp","version":1,"field":"create_time"},{"type":"int64","optional":true,"name":"io.debezium.time.Timestamp","version":1,"field":"update_time"},{"type":"int64","optional":true,"field":"agent_code"},{"type":"string","optional":true,"field":"agent_account"},{"type":"string","optional":true,"field":"play_method"},{"type":"string","optional":true,"field":"number_detail"},{"type":"int64","optional":true,"field":"develop_id"},{"type":"string","optional":true,"default":"","field":"bill_date"},{"type":"int32","optional":true,"default":0,"field":"cancel_cash_times"},{"type":"string","optional":true,"default":"0","field":"issue_num"},{"type":"double","optional":true,"default":0.0,"field":"min_odds"},{"type":"string","optional":true,"default":"","field":"register_from"},{"type":"string","optional":true,"default":"","field":"login_channel"},{"type":"int64","optional":true,"default":0,"field":"main_amount"},{"type":"int64","optional":true,"default":0,"field":"redpackets_amount"},{"type":"int64","optional":true,"default":0,"field":"activity_amount"},{"type":"int64","optional":true,"default":0,"field":"activity_id"},{"type":"string","optional":true,"default":"0","field":"redpackets_detail"},{"type":"string","optional":true,"default":"0","field":"activity_type"},{"type":"int16","optional":true,"default":0,"field":"settle_status"},{"type":"int64","optional":true,"default":0,"field":"settle_amount"},{"type":"int64","optional":true,"default":0,"field":"wh_tax"},{"type":"int64","optional":true,"name":"io.debezium.time.Timestamp","version":1,"field":"settle_time"},{"type":"string","optional":false,"field":"db_name"},{"type":"string","optional":false,"field":"table_name"},{"type":"int32","optional":false,"field":"dml_type"}],"optional":false,"name":"ng_g_game_deal_order.Value"},"payload":{"order_id":"8773639782452027392102012759","out_order_id":"t8773639782411239424","pay_order_id":"1020127598773639782501179392","order_status":1,"result_detail":"","game_id":60015,"game_name":"Gold Hunter","game_type":2,"user_id":102012759,"user_account":"1730000255","account_code":10201275902416582,"device_code":"3bd7151f-0bf3-404f-bf02-119c749645ec","platform":3,"payable_amount":5000000,"receipts_amount":5000000,"discount_amount":0,"recovery_amount":0,"coupon_type":"","excise_tax":0,"currency":"NGN","coupon_id":null,"bet_detail":"","description":null,"cancel_time":null,"create_time":1786008944000,"update_time":null,"agent_code":null,"agent_account":null,"play_method":null,"number_detail":"1;1;5000000;|","develop_id":10,"bill_date":"20260806","cancel_cash_times":0,"issue_num":"0","min_odds":0.0,"register_from":"ilot","login_channel":"3","main_amount":5000000,"redpackets_amount":0,"activity_amount":0,"activity_id":0,"redpackets_detail":"0","activity_type":"0","settle_status":0,"settle_amount":0,"wh_tax":0,"settle_time":null,"db_name":"ilot_trade1","table_name":"g_game_deal_order_0","dml_type":1}}
{"schema":{"type":"struct","fields":[{"type":"string","optional":false,"field":"order_id"},{"type":"string","optional":true,"default":"","field":"out_order_id"},{"type":"string","optional":true,"default":"","field":"pay_order_id"},{"type":"int16","optional":true,"field":"order_status"},{"type":"string","optional":true,"default":"","field":"result_detail"},{"type":"int32","optional":false,"default":0,"field":"game_id"},{"type":"string","optional":true,"field":"game_name"},{"type":"int32","optional":true,"default":0,"field":"game_type"},{"type":"int64","optional":false,"field":"user_id"},{"type":"string","optional":true,"field":"user_account"},{"type":"int64","optional":true,"field":"account_code"},{"type":"string","optional":true,"default":"","field":"device_code"},{"type":"int16","optional":true,"default":0,"field":"platform"},{"type":"int64","optional":false,"default":0,"field":"payable_amount"},{"type":"int64","optional":false,"default":0,"field":"receipts_amount"},{"type":"int64","optional":false,"default":0,"field":"discount_amount"},{"type":"int64","optional":true,"default":0,"field":"recovery_amount"},{"type":"string","optional":true,"default":"","field":"coupon_type"},{"type":"int64","optional":true,"default":0,"field":"excise_tax"},{"type":"string","optional":true,"default":"","field":"currency"},{"type":"int64","optional":true,"field":"coupon_id"},{"type":"string","optional":true,"default":"","field":"bet_detail"},{"type":"string","optional":true,"field":"description"},{"type":"int64","optional":true,"name":"io.debezium.time.Timestamp","version":1,"field":"cancel_time"},{"type":"int64","optional":true,"name":"io.debezium.time.Timestamp","version":1,"field":"create_time"},{"type":"int64","optional":true,"name":"io.debezium.time.Timestamp","version":1,"field":"update_time"},{"type":"int64","optional":true,"field":"agent_code"},{"type":"string","optional":true,"field":"agent_account"},{"type":"string","optional":true,"field":"play_method"},{"type":"string","optional":true,"field":"number_detail"},{"type":"int64","optional":true,"field":"develop_id"},{"type":"string","optional":true,"default":"","field":"bill_date"},{"type":"int32","optional":true,"default":0,"field":"cancel_cash_times"},{"type":"string","optional":true,"default":"0","field":"issue_num"},{"type":"double","optional":true,"default":0.0,"field":"min_odds"},{"type":"string","optional":true,"default":"","field":"register_from"},{"type":"string","optional":true,"default":"","field":"login_channel"},{"type":"int64","optional":true,"default":0,"field":"main_amount"},{"type":"int64","optional":true,"default":0,"field":"redpackets_amount"},{"type":"int64","optional":true,"default":0,"field":"activity_amount"},{"type":"int64","optional":true,"default":0,"field":"activity_id"},{"type":"string","optional":true,"default":"0","field":"redpackets_detail"},{"type":"string","optional":true,"default":"0","field":"activity_type"},{"type":"int16","optional":true,"default":0,"field":"settle_status"},{"type":"int64","optional":true,"default":0,"field":"settle_amount"},{"type":"int64","optional":true,"default":0,"field":"wh_tax"},{"type":"int64","optional":true,"name":"io.debezium.time.Timestamp","version":1,"field":"settle_time"},{"type":"string","optional":false,"field":"db_name"},{"type":"string","optional":false,"field":"table_name"},{"type":"int32","optional":false,"field":"dml_type"}],"optional":false,"name":"ng_g_game_deal_order.Value"},"payload":{"order_id":"8773639782452027392102012759","out_order_id":"t8773639782411239424","pay_order_id":"1020127598773639782501179392","order_status":1,"result_detail":"","game_id":60015,"game_name":"Gold Hunter","game_type":2,"user_id":102012759,"user_account":"1730000255","account_code":10201275902416582,"device_code":"3bd7151f-0bf3-404f-bf02-119c749645ec","platform":3,"payable_amount":5000000,"receipts_amount":5000000,"discount_amount":0,"recovery_amount":0,"coupon_type":"","excise_tax":0,"currency":"NGN","coupon_id":null,"bet_detail":"","description":null,"cancel_time":null,"create_time":1786008944000,"update_time":1786008956000,"agent_code":null,"agent_account":null,"play_method":null,"number_detail":"1;1;5000000;|","develop_id":10,"bill_date":"20260806","cancel_cash_times":0,"issue_num":"0","min_odds":0.0,"register_from":"ilot","login_channel":"3","main_amount":5000000,"redpackets_amount":0,"activity_amount":0,"activity_id":0,"redpackets_detail":"0","activity_type":"0","settle_status":3,"settle_amount":7500000,"wh_tax":0,"settle_time":1786008956000,"db_name":"ilot_trade1","table_name":"g_game_deal_order_0","dml_type":2}}

### Solution

_No response_

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the partitioning attempts on Paimon 1.4.2 or master with Flink 1.20 and paimon-kafka-connect, using create_time from the Debezium JSON records. Trace the validation of partition expressions and schema handling. Done means the requested date-based partitioning behavior is supported and verified with the provided create_time examples.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka, sql
Domain
data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.