[SUPPORT] how to config hudi table TTL in S3? The table_meta can be separated into a directory?
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
How to configure TTL policy in hudi data table? Can the metadata (.hoodie )be separated into a directory?
Only configure the appropriate TTL for the data directory, so that data cleaning can also use hierarchical storage and different life cycles, and the data can also be automatically cleaned by relying on the object storage service, and there is no cost.
EG:
--------------------------------------------------------------------
s3://big-data-eu/hudi/data/bi_ods/table_name/dt=20231211/data
< 30 days STANDARD S3
> 30 days delete by TTL with no cost.
=================================
< 15 days STANDARD S3
> 15 days GLACIER_IR
> 105 days delete by TTL with no cost.
s3://big-data-eu/hudi/table_meta/bi_ods/table_name/.hoodie
As mentioned above, if there are many data tables under the data storage and the storage periods are the same, I can just configure the storage period for the directory and rely on the object storage to automatically clean up the historical data at no cost.
EG:
--------------------------------------------------------------------
s3://big-data-eu/hudi/data/30days/bi_ods/table_name/dt=20231211/data
< 30 days STANDARD S3
> 30 days delete by TTL with no cost.
s3://big-data-eu/hudi/data/90days/bi_ods/table_name/dt=20231211/data
< 30 days STANDARD S3
30days < 90days GLACIER_IR
> 90 days delete by TTL with no cost.
....
=================================
< 15 days STANDARD S3
> 15 days GLACIER_IR
> 105 days delete by TTL with no cost.
s3://big-data-eu/hudi/table_meta/bi_ods/table_name/.hoodie
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.