apache / apache/hudi

Add record count payload to support pv/uv

Open
#15,638 1 comment 0 reactions 0 assignees View on GitHub
from-jira priority:high status:pr-available type:feature
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

In the past, pv/uv was processed through flink + window aggregation. This method has the risk of delayed data discarding and state explosion. We use record count payload without these risks.

In order to use 'RecordCountAvroPayload', we need to add field [hoodie_record_count bigint] to the schema when creating the hudi table to record the result of pv/uv, field 'hoodie_record_count' does not need to be filled, and flink will automatically set it to "null", "null" represents 1

eg:
Order field is 'ts', schema is :

{[    

{"name":"id","type":"string"}

,

    \{"name":"ts","type":"long"},

    \{"name":"name","type":"string"},

    \{"name":"hoodie_record_count","type":"long"}

]}

case 1
Current data:
id   ts   name       hoodie_record_count
 1    1    name_1     1
Insert data:
id   ts    name      hoodie_record_count
 1    2    name_2   2

Result data:
id   ts    name       hoodie_record_count
 1    2     name_2   3

case 2
Current data:
id   ts    name      hoodie_record_count
1    2      name_1   null
Insert data:
id   ts    name      hoodie_record_count
 1    1     name_2   1

Result data:
id   ts    name      hoodie_record_count
 1    2     name_1   2

 

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-5413
- Type: New Feature

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the RecordCountAvroPayload reference and the linked JIRA issue HUDI-5413; no source files or tests are named in the report. Verify where the Hudi table schema is created and how Flink supplies the payload. Done means schemas include hoodie_record_count and the two described aggregation cases produce the expected counts.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, stream-processing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.