GoogleContainerTools / GoogleContainerTools/skaffold

Speed up deployments by rendering and deploying concurrently

Open
#8,363 16 comments 29 reactions 0 assignees View on GitHub
kind/feature-request priority/p3
Dominant language
Go
Stars
15.9k
Forks
1.7k
Avg merge
3d 9h
Merged PRs (30d)
10

Description

Most of our time is spent **waiting** for helm chart render and deployment to run one by one (this is exceptionally problematic when deploying/rendering tasks is taking a few minutes).
We could significally **reduce** deployment time by processing the `required skaffold files` in **parallel**.

### Expected behavior
We expect skaffold to render and deploy tasks in **parallel**.

### Actual behavior
Each helm chart is rendered and deployed **one by one**..

### Information

- Skaffold version: v2.1.0
- Operating system: macOS 13.1 (22C65)
- Installed via: Homebrew
- Contents of skaffold.yaml:

```yaml
apiVersion: skaffold/v3
kind: Config
requires:
- path: ./app1/skaffold.yml
- path: ./app2/skaffold.yml
- path: ./app3/skaffold.yml
- path: ./app4/skaffold.yml
- path: ./app5/skaffold.yml

...
```

### Steps to reproduce the behavior

1. `git clone https://github.com/iosifnicolae2/skaffold-bug`
2. `cd skaffold-bug/skaffold`
3. `skaffold run --verbosity debug` (you migth need to update `build.artifacts.image` from each `skaffold.yml` files)
4. As you can see the `Render` and `Deploy` tasks are synchronius.
```
Running command: [helm --kube-context cluster.local template app2 skaffold-bug/charts/app -
Running command: [helm --kube-context cluster.local template app2 skaffold-bug/charts/app --post-renderer /opt/homebrew/bin/skaffold --set image.repository=bringes/app1 --set image.tag=XXXXX] subtask=1 task=Render-post-renderer /opt/homebrew/bin/skaffold --set image.repository=bringes/app1 --set image.tag=XXXXX] subtask=2 task=Render
Running command: [helm --kube-context cluster.local template app2 skaffold-bug/charts/app --post-renderer /opt/homebrew/bin/skaffold --set image.repository=bringes/app1 --set image.tag=XXXXX] subtask=3 task=Render
...

Running command: [helm --kube-context cluster.local dep build skaffold-bug/charts/app] subtask=1 task=Deploy
Running command: [helm --kube-context cluster.local dep build skaffold-bug/charts/app] subtask=2 task=Deploy
Running command: [helm --kube-context cluster.local dep build skaffold-bug/charts/app] subtask=3 task=Deploy
...

```

It might be perfect to have an option to group certain requirements in a group which will be deployed concurrently , something like:
```yaml
apiVersion: skaffold/v3
kind: Config
requires:
- concurrency: 3
paths:
- ./app1/skaffold.yml
- ./app2/skaffold.yml
- ./app3/skaffold.yml
- ./app4/skaffold.yml
- path: ./app5/skaffold.yml

```

**Obs!** It's pretty important to be able to deploy certain requirements **synchronously**.
**Obs!** For rendering the helm charts, maybe we could execute multiple tasks **in parallel** by default..

Similar with https://github.com/GoogleContainerTools/skaffold/issues/5417

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.