apache / apache/hamilton

Hamilton UI - Configuration menu for clean up and tracked execution infos

Open
#1,228 7 comments 0 reactions 0 assignees View on GitHub
SDK UI
Dominant language
Jupyter Notebook
Stars
2.6k
Forks
213
PR merge metrics
No merged PRs in 30d

Description

Updated issue:

1. A user cannot delete data easily from the Hamilton UI without going to the database manually.

Proposed solution:

* Expose server endpoint and UI view to delete projects, and runs; expose SDK functionality to do it programmatically.

Alternatives:

* #1232 helps mitigate the problem by enabling more configuration driven options to determine what is or is not stored.

---- Original Issue: ---
**Is your feature request related to a problem? Please describe.**
Currently, the Hamilton Tracker/UI saves a lot of information for every dataflow run, like success info, node parameters (inputs), node results(outputs), execution times,...

For ETL dataflows, that runs often and processes a lot of data during every run, this results in a lot of data stored in the postgres/sqlite db (might be a problem) and the RUNS section in the UI becomes unresponsive.

Personally, I am most interested in the success info, execution times and logs.

**Describe the solution you'd like**
1. Configuration of regular cleanups for the tracker data of each run. Ideally, this can be done individual for every data type (success info, inputs, outputs, logs,...)
3. Configuration of which kind of data should be stored by the tracker.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.