Autodesk / Autodesk/shore

Parallelize Nested Pipeline save

Open
#59 0 comments 0 reactions 0 assignees View on GitHub
Backend-Spinnaker enhancement save
Dominant language
Go
Stars
21
Forks
4
PR merge metrics
No merged PRs in 30d

Description

Current implementation of the nested pipeline save uses a naive DFS implementation.

Meaning, each new pipeline stage & pipeline is saved sequentially.

For highly nested pipelines or pipelines with many stages this sequential save could cause performance issues.

To mitigate this problem, we should use a BFS approach and save pipelines siblings concurrently. This will mitigate the performance problems to some extent.

In addition, it might be wise to add logging in case of very nested pipelines (Depth > 3) to notify the developers of a possible issue.

Contributor guide

No contributing guide indexed for this repository

Research direction

Locate the current nested pipeline save implementation and trace its depth-first traversal and sequential stage saves. Evaluate a breadth-first approach that saves sibling pipelines concurrently, add the requested notification for depth greater than 3, and verify that nested pipelines still save correctly.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
performance
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.