Add record count payload to support pv/uv
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
In the past, pv/uv was processed through flink + window aggregation. This method has the risk of delayed data discarding and state explosion. We use record count payload without these risks.
In order to use 'RecordCountAvroPayload', we need to add field [hoodie_record_count bigint] to the schema when creating the hudi table to record the result of pv/uv, field 'hoodie_record_count' does not need to be filled, and flink will automatically set it to "null", "null" represents 1
eg:
Order field is 'ts', schema is :
{[
{"name":"id","type":"string"}
,
\{"name":"ts","type":"long"},
\{"name":"name","type":"string"},
\{"name":"hoodie_record_count","type":"long"}
]}
case 1
Current data:
id ts name hoodie_record_count
1 1 name_1 1
Insert data:
id ts name hoodie_record_count
1 2 name_2 2
Result data:
id ts name hoodie_record_count
1 2 name_2 3
case 2
Current data:
id ts name hoodie_record_count
1 2 name_1 null
Insert data:
id ts name hoodie_record_count
1 1 name_2 1
Result data:
id ts name hoodie_record_count
1 2 name_1 2
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-5413
- Type: New Feature
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the RecordCountAvroPayload reference and the linked JIRA issue HUDI-5413; no source files or tests are named in the report. Verify where the Hudi table schema is created and how Flink supplies the payload. Done means schemas include hoodie_record_count and the two described aggregation cases produce the expected counts.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, stream-processing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100