apache / apache/burr

Configurable Persistence Frequency

Open
#469 2 comments 0 reactions 0 assignees View on GitHub
area/storage kind/feature priority/medium
Dominant language
Python
Stars
2.5k
Forks
195
Avg merge
6d 10h
Merged PRs (30d)
17

Description

**Is your feature request related to a problem? Please describe.**
Currently we persist after every node. We should allow this to be configurable. This saves you from an over-saving state, which can cause slowness/state bloat.

**Describe the solution you'd like**
A few ideas:
1. Add a filter in the `with_persister` argument -- probably node names (`exclude`/`include`)
2. Add in the Action itself (E.G. have an action say if it should be persisted/tracked/transient)
3. Integrate with tags (feature we don't have yet) -- have a `tags=['no_persist']` or something -- see https://github.com/DAGWorks-Inc/burr/issues/468

So really, do we want this at the action or application level? My instinct is that we can have both but the action level is more valuable, need to think it through though...

**Describe alternatives you've considered**
See above + https://github.com/DAGWorks-Inc/burr/issues/468.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.