argoproj / argoproj/argo-workflows

status.storedtemplates field content should be offloaded to database

Open
#14,716 8 comments 2 reactions 0 assignees View on GitHub
area/offloading
Dominant language
Go
Stars
17k
Forks
3.7k
Avg merge
1d 20h
Merged PRs (30d)
138

Description

### Pre-requisites

- [x] I have double-checked my configuration
- [x] I have tested with the `:latest` image tag (i.e. `quay.io/argoproj/workflow-controller:latest`) and can confirm the issue still exists on `:latest`. If not, I have explained why, **in detail**, in my description below.
- [x] I have searched existing issues and could not find a match for this bug
- [x] I'd like to contribute the fix myself (see [contributing guide](https://github.com/argoproj/argo-workflows/blob/main/docs/CONTRIBUTING.md))

### What happened? What did you expect to happen?

HI, I'm using Argo Workflow as my workflow execution engine, but I've noticed that when a `workflow` resource references many `Templates` from `WorkflowTemplates`, the `workflow` resource quickly hits etcd's default request body size limit (1MB). I previously learned that `status.node` is stored in a database, so I believe `status.storedTemplates` could also be stored in the database.

So, do we have a plan to solve this problem at present?

### Version(s)

v3.5.7

### Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflow that uses private images.

```YAML
1. create 1000 workflowtemplates likes

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
name: test-pipeline-{{i}}
namespace: mlplatform-pipeline
spec:
arguments: {}
entrypoint: CustomTask
templates:
- inputs: {}
metadata: {}
name: test-pipeline-{{i}}
outputs:
parameters:
- name: output_id
valueFrom:
supplied: {}
plugin:
pipeline-http-plugin:
action: create
body:
testdata: balabalabalabalabalabalabalabalabalabalabalabalabalabalabala...balabalabalabalabalabalabalabalabalabalabalabalabalabalabala(1MB size)

2. create a workflow which references these workflowtemplates

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
name: test
namespace: test
spec:
arguments: {}
entrypoint: main
templates:
- dag:
tasks:
- arguments: {}
name: test-pipeline
templateRef:
name: test-pipeline
template: test-pipeline
- arguments:
dependencies:
- test-pipeline
name: test-pipeline2
templateRef:
name: test-pipeline2
template: test-pipeline2
...
- arguments:
dependencies:
- test-pipeline
name: test-pipeline1000
templateRef:
name: test-pipeline1000
template: test-pipeline1000
inputs: {}
metadata: {}
name: main
outputs: {}
```

### Logs from the workflow controller

```text
kubectl logs -n argo deploy/workflow-controller | grep ${workflow}
```

### Logs from in your workflow's wait container

```text
kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded
```

Contributor guide

Open the contributing guide

Research direction

Start by tracing how status.storedTemplates is persisted and compare it with the existing status.node database handling. Use the provided large WorkflowTemplate reproduction to verify the change; done means workflows referencing many templates no longer hit etcd's 1 MB request-body limit while stored templates remain available.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
backend, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.