apache / apache/hudi

Add Call add_ttl_policy for spark sql

Open
#15,626 0 comments 0 reactions 0 assignees View on GitHub
from-jira priority:high type:feature
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

# Data management by partition:  During partition clearing, if the number of partitions exceeds the configured number, delete older partitions until the number of partitions equals the configured maximum number.
# Management by data volume: During partition clearing, the system collects statistics on the total data volume. If the data volume exceeds the specified value, the system deletes the earliest partition until the total data volume of the partition is less than or equal to the specified value
# Management by expiration time: During partition clearing, the system calculates the expiration time based on the last modified time of the partition and deletes the expired partition.

!image-2022-12-13-17-34-26-346.png!

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-5379
- Type: New Feature
- Attachment(s):
- 13/Dec/22 09:34;melin;image-2022-12-13-17-34-26-346.png;https://issues.apache.org/jira/secure/attachment/13053825/image-2022-12-13-17-34-26-346.png

Contributor guide

No contributing guide indexed for this repository

Research direction

The issue names Spark SQL and the HUDI-5379 JIRA request but identifies no source files, tests, or entry points. Start by locating the Spark SQL add_ttl_policy integration and the partition-clearing implementation; done means the three described limits are enforced.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.