Row Level TTL Support for records stored in Hudi

For e:g : Have records only updated last month 

 

GH: https://github.com/apache/hudi/issues/2743

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-1741
- Type: New Feature

---

## Comments

31/Mar/21 00:43;vbalaji;[~shivnarayan] : FYI;;;

---

03/Apr/21 16:28;pratyakshsharma;Guess the same can be handled with this Jira - https://issues.apache.org/jira/browse/HUDI-349? [~vbalaji] [~shivnarayan];;;

---

05/Apr/21 15:25;aditiwari;[~pratyakshsharma] I guess with time based cleaning policy, we might need some modifications in compactor as well. 

For a recently updated base file also some of its records might be older.

Time based cleaner and filtering out records with older commit time while compacting(in MOR) or rewriting(in COW) base file should solve the issue.;;;

---

28/Oct/22 03:09;nicholasjiang;[~shivnarayan], IMO, each record of hudi has the commit time of hudi. The solution is to first follow the TTL, do not display expired data when checking, or even push down to the data source directly, and then delete it when doing operations such as clustering that need to rewrite the data. WDYT?

cc [~xleesf] ;;;

---

28/Oct/22 03:29;xleesf;[~nicholasjiang] agree with the solution;;;

Start by reviewing the Hudi-1741 and Hudi-349 references, then trace the time-based cleaner, compactor, MOR/COW rewriting, and clustering paths mentioned in the discussion. Define how row-level TTL should hide expired records and remove them during rewrite operations, and add coverage for records that are old within a recently updated base file.

Written by the indexing model from the issue text.