Add Call add_ttl_policy for spark sql
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
# Data management by partition: During partition clearing, if the number of partitions exceeds the configured number, delete older partitions until the number of partitions equals the configured maximum number.
# Management by data volume: During partition clearing, the system collects statistics on the total data volume. If the data volume exceeds the specified value, the system deletes the earliest partition until the total data volume of the partition is less than or equal to the specified value
# Management by expiration time: During partition clearing, the system calculates the expiration time based on the last modified time of the partition and deletes the expired partition.
!image-2022-12-13-17-34-26-346.png!
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-5379
- Type: New Feature
- Attachment(s):
- 13/Dec/22 09:34;melin;image-2022-12-13-17-34-26-346.png;https://issues.apache.org/jira/secure/attachment/13053825/image-2022-12-13-17-34-26-346.png
Contributor guide
No contributing guide indexed for this repository
Research direction
The issue names Spark SQL and the HUDI-5379 JIRA request but identifies no source files, tests, or entry points. Start by locating the Spark SQL add_ttl_policy integration and the partition-clearing implementation; done means the three described limits are enforced.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100