argoproj / argoproj/argo-workflows
status.storedtemplates field content should be offloaded to database
- Dominant language
- Go
- Stars
- 17k
- Forks
- 3.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 138
Description
### Pre-requisites
- [x] I have double-checked my configuration
- [x] I have tested with the `:latest` image tag (i.e. `quay.io/argoproj/workflow-controller:latest`) and can confirm the issue still exists on `:latest`. If not, I have explained why, **in detail**, in my description below.
- [x] I have searched existing issues and could not find a match for this bug
- [x] I'd like to contribute the fix myself (see [contributing guide](https://github.com/argoproj/argo-workflows/blob/main/docs/CONTRIBUTING.md))
### What happened? What did you expect to happen?
HI, I'm using Argo Workflow as my workflow execution engine, but I've noticed that when a `workflow` resource references many `Templates` from `WorkflowTemplates`, the `workflow` resource quickly hits etcd's default request body size limit (1MB). I previously learned that `status.node` is stored in a database, so I believe `status.storedTemplates` could also be stored in the database.
So, do we have a plan to solve this problem at present?
### Version(s)
v3.5.7
### Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflow that uses private images.
```YAML
1. create 1000 workflowtemplates likes
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
name: test-pipeline-{{i}}
namespace: mlplatform-pipeline
spec:
arguments: {}
entrypoint: CustomTask
templates:
- inputs: {}
metadata: {}
name: test-pipeline-{{i}}
outputs:
parameters:
- name: output_id
valueFrom:
supplied: {}
plugin:
pipeline-http-plugin:
action: create
body:
testdata: balabalabalabalabalabalabalabalabalabalabalabalabalabalabala...balabalabalabalabalabalabalabalabalabalabalabalabalabalabala(1MB size)
2. create a workflow which references these workflowtemplates
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
name: test
namespace: test
spec:
arguments: {}
entrypoint: main
templates:
- dag:
tasks:
- arguments: {}
name: test-pipeline
templateRef:
name: test-pipeline
template: test-pipeline
- arguments:
dependencies:
- test-pipeline
name: test-pipeline2
templateRef:
name: test-pipeline2
template: test-pipeline2
...
- arguments:
dependencies:
- test-pipeline
name: test-pipeline1000
templateRef:
name: test-pipeline1000
template: test-pipeline1000
inputs: {}
metadata: {}
name: main
outputs: {}
```
### Logs from the workflow controller
```text
kubectl logs -n argo deploy/workflow-controller | grep ${workflow}
```
### Logs from in your workflow's wait container
```text
kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded
```
Contributor guide
Research direction
Start by tracing how status.storedTemplates is persisted and compare it with the existing status.node database handling. Use the provided large WorkflowTemplate reproduction to verify the change; done means workflows referencing many templates no longer hit etcd's 1 MB request-body limit while stored templates remain available.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- backend, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100