apache / apache/hudi

[SUPPORT] how to config hudi table TTL in S3? The table_meta can be separated into a directory?

Open
#10,316 4 comments 0 reactions 1 assignee Claimed by @stream2000 View on GitHub
priority:medium type:feature
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

How to configure TTL policy in hudi data table? Can the metadata (.hoodie )be separated into a directory?
Only configure the appropriate TTL for the data directory, so that data cleaning can also use hierarchical storage and different life cycles, and the data can also be automatically cleaned by relying on the object storage service, and there is no cost.

EG:
--------------------------------------------------------------------
s3://big-data-eu/hudi/data/bi_ods/table_name/dt=20231211/data
< 30 days STANDARD S3
> 30 days delete by TTL with no cost.
=================================

< 15 days STANDARD S3
> 15 days GLACIER_IR
> 105 days delete by TTL with no cost.

s3://big-data-eu/hudi/table_meta/bi_ods/table_name/.hoodie

As mentioned above, if there are many data tables under the data storage and the storage periods are the same, I can just configure the storage period for the directory and rely on the object storage to automatically clean up the historical data at no cost.
EG:
--------------------------------------------------------------------
s3://big-data-eu/hudi/data/30days/bi_ods/table_name/dt=20231211/data
< 30 days STANDARD S3
> 30 days delete by TTL with no cost.

s3://big-data-eu/hudi/data/90days/bi_ods/table_name/dt=20231211/data
< 30 days STANDARD S3
30days < 90days GLACIER_IR
> 90 days delete by TTL with no cost.
....
=================================

< 15 days STANDARD S3
> 15 days GLACIER_IR
> 105 days delete by TTL with no cost.

s3://big-data-eu/hudi/table_meta/bi_ods/table_name/.hoodie

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.