apache / apache/burr

Umbrella issue -- parallelism improvements

Open
#427 1 comment 0 reactions 0 assignees View on GitHub
area/core area/streaming kind/umbrella lifecycle/frozen priority/medium
Dominant language
Python
Stars
2.5k
Forks
195
Avg merge
6d 10h
Merged PRs (30d)
17

Description

All the improvements to parallelism that we want to make post the first release:

- [ ] UI improvements
- [x] Expand parallel apps inline (#421)
- [x] Show success/failure
- [ ] Ensure insights tab shows aggregation of all parallel sub-applications
- [ ] Show the sub-applications graphs in the graph
- [x] Ensure live mode works (#421)
- [x] Ensure the hierarchy of traces is clear (#421)
- [ ] Task creation/management
- [ ] Add API for task IDs
- [ ] Add API
- [ ] Wire through to UI
- [ ] Add documentation
- [ ] Guard-rails around async problems
- [ ] Ensure that generators that are both async and not async work as expected
- [x] Wiring through persisters
- [x] Allow wiring through of persisters to sub-trackers
- [x] Implement this spec: https://burr.dagworks.io/concepts/parallelism/#persistence-tracking -- not currently doable
- [ ] Execution improvements
- [ ] Reevaluate execution API
- [ ] Add guard-rails for executors (E.G. with async, etc...)
- [ ] Allow customization of execution capabilities on a per-task basis
- [x] Ensure recursion works -- multiple levels deep

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.