Parallelize Nested Pipeline save
- Dominant language
- Go
- Stars
- 21
- Forks
- 4
- PR merge metrics
- No merged PRs in 30d
Description
Current implementation of the nested pipeline save uses a naive DFS implementation.
Meaning, each new pipeline stage & pipeline is saved sequentially.
For highly nested pipelines or pipelines with many stages this sequential save could cause performance issues.
To mitigate this problem, we should use a BFS approach and save pipelines siblings concurrently. This will mitigate the performance problems to some extent.
In addition, it might be wise to add logging in case of very nested pipelines (Depth > 3) to notify the developers of a possible issue.
Contributor guide
No contributing guide indexed for this repository
Research direction
Locate the current nested pipeline save implementation and trace its depth-first traversal and sequential stage saves. Evaluate a breadth-first approach that saves sibling pipelines concurrently, add the requested notification for depth greater than 3, and verify that nested pipelines still save correctly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- performance
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100