GoogleContainerTools / GoogleContainerTools/skaffold
Speed up deployments by rendering and deploying concurrently
- Dominant language
- Go
- Stars
- 15.9k
- Forks
- 1.7k
- Avg merge
- 3d 9h
- Merged PRs (30d)
- 10
Description
Most of our time is spent **waiting** for helm chart render and deployment to run one by one (this is exceptionally problematic when deploying/rendering tasks is taking a few minutes).
We could significally **reduce** deployment time by processing the `required skaffold files` in **parallel**.
### Expected behavior
We expect skaffold to render and deploy tasks in **parallel**.
### Actual behavior
Each helm chart is rendered and deployed **one by one**..
### Information
- Skaffold version: v2.1.0
- Operating system: macOS 13.1 (22C65)
- Installed via: Homebrew
- Contents of skaffold.yaml:
```yaml
apiVersion: skaffold/v3
kind: Config
requires:
- path: ./app1/skaffold.yml
- path: ./app2/skaffold.yml
- path: ./app3/skaffold.yml
- path: ./app4/skaffold.yml
- path: ./app5/skaffold.yml
...
```
### Steps to reproduce the behavior
1. `git clone https://github.com/iosifnicolae2/skaffold-bug`
2. `cd skaffold-bug/skaffold`
3. `skaffold run --verbosity debug` (you migth need to update `build.artifacts.image` from each `skaffold.yml` files)
4. As you can see the `Render` and `Deploy` tasks are synchronius.
```
Running command: [helm --kube-context cluster.local template app2 skaffold-bug/charts/app -
Running command: [helm --kube-context cluster.local template app2 skaffold-bug/charts/app --post-renderer /opt/homebrew/bin/skaffold --set image.repository=bringes/app1 --set image.tag=XXXXX] subtask=1 task=Render-post-renderer /opt/homebrew/bin/skaffold --set image.repository=bringes/app1 --set image.tag=XXXXX] subtask=2 task=Render
Running command: [helm --kube-context cluster.local template app2 skaffold-bug/charts/app --post-renderer /opt/homebrew/bin/skaffold --set image.repository=bringes/app1 --set image.tag=XXXXX] subtask=3 task=Render
...
Running command: [helm --kube-context cluster.local dep build skaffold-bug/charts/app] subtask=1 task=Deploy
Running command: [helm --kube-context cluster.local dep build skaffold-bug/charts/app] subtask=2 task=Deploy
Running command: [helm --kube-context cluster.local dep build skaffold-bug/charts/app] subtask=3 task=Deploy
...
```
It might be perfect to have an option to group certain requirements in a group which will be deployed concurrently , something like:
```yaml
apiVersion: skaffold/v3
kind: Config
requires:
- concurrency: 3
paths:
- ./app1/skaffold.yml
- ./app2/skaffold.yml
- ./app3/skaffold.yml
- ./app4/skaffold.yml
- path: ./app5/skaffold.yml
```
**Obs!** It's pretty important to be able to deploy certain requirements **synchronously**.
**Obs!** For rendering the helm charts, maybe we could execute multiple tasks **in parallel** by default..
Similar with https://github.com/GoogleContainerTools/skaffold/issues/5417
Contributor guide
Assessment
This issue has not been assessed yet.