apache / apache/incubator-xtable

Add a proposal to expiry and cleanup artifacts generated by XTable

Open
#655 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
1.2k
Forks
212
Avg merge
4d 9h
Merged PRs (30d)
16

Description

Develop a proposal to manage artifacts generated by XTable, focusing on efficient expiry and cleanup of files generated by XTable for storage used by Target format and query performance.

XTable generates `Commit Logs` which are typically small in size. It will soon start generating `Deletion Files` which are expected to grow quickly and require more storage management.

Cleanup Policies could align artifact cleanup with the existing policies of the source table, or be implemented using specific configurations within XTable.

Potential topics to discuss include policy management, automated and custom cleanup mechanism, and monitoring and reporting.

_Originally posted by @the-other-tim-brown in https://github.com/apache/incubator-xtable/pull/634#discussion_r1956695495_

Contributor guide

No contributing guide indexed for this repository

Research direction

No files or tests are named. Start by reviewing XTable's Commit Logs and planned Deletion Files, then compare source-table cleanup policies with XTable-specific configuration. Done means a documented proposal covering policy management, automated and custom cleanup, monitoring, and reporting.

Written by the indexing model from the issue text.

Assessment

Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.