flyteorg / flyteorg/flyte

[Core feature] Support for Postgres database pruning / data retention

Open
#6,360 4 comments 1 reaction 1 assignee Claimed by @yuhuan130 View on GitHub
enhancement
Dominant language
Go
Stars
7.5k
Forks
886
Avg merge
1d 12h
Merged PRs (30d)
120

Description

### Motivation: Why do you think this is important?

As Flyte runs it will accumulate rows in the Postgres database. In order to keep Postgres disk usage under control it would be helpful if there was an option to configure database pruning or a data retention policy. This may also retain query performance over time.

### Goal: What should the final outcome look like, ideally?

I imagine there could be a pruning go binary that could run as a Kubernetes cron job that would deleter executions (or other items) if they are in a terminal state and have an age over a configurable threshold.

### Describe alternatives you've considered

None yet

### Propose: Link/Inline OR Additional context

_No response_

### Are you sure this issue hasn't been raised already?

- [x] Yes

### Have you read the Code of Conduct?

- [x] Yes

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.