apache / apache/incubator-xtable
Add a proposal to expiry and cleanup artifacts generated by XTable
- Dominant language
- Java
- Stars
- 1.2k
- Forks
- 212
- Avg merge
- 4d 9h
- Merged PRs (30d)
- 16
Description
Develop a proposal to manage artifacts generated by XTable, focusing on efficient expiry and cleanup of files generated by XTable for storage used by Target format and query performance.
XTable generates `Commit Logs` which are typically small in size. It will soon start generating `Deletion Files` which are expected to grow quickly and require more storage management.
Cleanup Policies could align artifact cleanup with the existing policies of the source table, or be implemented using specific configurations within XTable.
Potential topics to discuss include policy management, automated and custom cleanup mechanism, and monitoring and reporting.
_Originally posted by @the-other-tim-brown in https://github.com/apache/incubator-xtable/pull/634#discussion_r1956695495_
Contributor guide
No contributing guide indexed for this repository
Research direction
No files or tests are named. Start by reviewing XTable's Commit Logs and planned Deletion Files, then compare source-table cleanup policies with XTable-specific configuration. Done means a documented proposal covering policy management, automated and custom cleanup, monitoring, and reporting.
Written by the indexing model from the issue text.
Assessment
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100