argoproj / argoproj/argo-workflows

OffloadNodeStatusRepo.List fetches all offloaded node blobs for a namespace with no pagination/limit, causing MySQL max_allowed_packet failures on ListWorkflows

Open
#16,611 3 comments 1 reaction 0 assignees View on GitHub
Dominant language
Go
Stars
17k
Forks
3.7k
Avg merge
1d 20h
Merged PRs (30d)
138

Description

### Pre-requisites

- [x] I have double-checked my configuration
- [x] I have tested with the `:latest` image tag (i.e. `quay.io/argoproj/workflow-controller:latest`) and can confirm the issue still exists on `:latest`. If not, I have explained why, **in detail**, in my description below.
- [x] I have searched existing issues and could not find a match for this bug
- [ ] I'd like to contribute the fix myself (see [contributing guide](https://github.com/argoproj/argo-workflows/blob/main/docs/CONTRIBUTING.md))

### What happened? What did you expect to happen?

### What happened/what did you expect to happen?

When `node status offloading` is enabled and a namespace has multiple concurrently
running workflows with large offloaded node status blobs (e.g. workflows with
thousands to tens of thousands of nodes), calling `ListWorkflows` fails because
the server attempts to fetch *all* offloaded node blobs for the namespace in a
single unbounded SQL query, with no per-workflow filtering, pagination, or limit.

This is functionally the same class of issue that was fixed for the *archived*
workflow list endpoint in #12025 / #13295 / #13601 (where the fix was to push
filtering down into SQL via JSON_EXTRACT / Postgres JSON operators instead of
loading full JSON blobs into memory) — but that fix does not appear to have been
applied to the **offloaded node status list path for live (non-archived) workflows**.

Specifically, `OffloadNodeStatusRepo.List()` in
`persist/sqldb/offload_node_status_repo.go` does:

​```go
func (wdc *nodeOffloadRepo) List(ctx context.Context, namespace string) (map[UUIDVersion]wfv1.Nodes, error) {
...
err := s.SQL().
Select("uid", "version", "nodes").
From(wdc.tableName).
Where(db.Cond{"clustername": wdc.clusterName}).
And(namespaceEqual(namespace)).
All(&records)
...
}
​```

This selects the full `nodes` JSON column for *every* offloaded live workflow in
the namespace, with no `LIMIT`/pagination, and no way to filter to only the
workflows actually needed for the current page of results. It's called from
`workflow_server.go`'s `ListWorkflows`:

​```go
if s.offloadNodeStatusRepo.IsEnabled() && !cleaner.WillExclude("items.status.nodes") {
offloadedNodes, err := s.offloadNodeStatusRepo.List(ctx, req.Namespace)
...
}
​```

If even a handful of workflows in a namespace have large node counts (e.g. 5-6
workflows each with 10K+ nodes), the combined result set of this query can
exceed MySQL's `max_allowed_packet` (default 64MB, and even with the limit
raised, this scales unboundedly with cluster usage), causing `ListWorkflows`
to fail entirely for the whole namespace — not just for the oversized workflows.

### Expected behavior

`ListWorkflows` should not require fetching offloaded node status for every
workflow in a namespace up front. Ideally:
- Filtering/pagination (namespace, phase, label selectors, limit/offset) should
be applied *before* fetching node status blobs, not after.
- Node status for workflows not part of the current page/result set should never
be fetched.
- As a stopgap, `OffloadNodeStatusRepo.List` could accept a set of workflow UIDs
to filter by, matching the UIDs already resolved from the live k8s workflow
list, instead of fetching the full namespace-wide record set.

### Workaround

Excluding `items.status.nodes` from the requested `fields` in the ListWorkflows
request (via `cleaner.WillExclude("items.status.nodes")`) skips this code path
entirely. This works if the caller doesn't need per-node status in list views,
but isn't viable for UIs/clients that show node-level progress in the list.

### Version(s)

v3.7.7 (also present on main branch as of check against
persist/sqldb/offload_node_status_repo.go)

### Environment

- MySQL as the persistence backend
- Node status offloading enabled
- Multiple concurrently running workflows with 10,000+ nodes each in the same namespace

### Related issues
- #12025 (same class of bug, but for the archived workflow list, already fixed)
- #13295 / #13601 (the fix pattern applied to archives — pushing filters into SQL)
- #13290 (broader "mysql write performance/stability" issue, related but distinct)

### Version(s)

v3.7.7

### Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflow that uses private images.

```YAML
{% set staticstr = "argo-scale-sw44057-retrystorm-" %}
{% set epoch_time = funcEpochTimeNS() %}
---
# SW-44057: Retry-storm load test.
#
# Reproduces (at a configurable scale) the SW-43993 incident pattern of many
# concurrent, high-retry-count tasks generating tens of thousands of failed
# attempts, to exercise workflow-controller concurrency (--workflow-workers)
# and persistence connection pool bounds under sustained reconciliation load.
#
# Incident reference: 394 Jobs (272 with backoffLimit: 9999), 51,978 total
# failed attempts, one target workflow alone accounting for 8,512 of them.
# Override RETRY_STORM_TASK_COUNT / RETRY_STORM_LIMIT to scale the
# reproduction to the target environment - see
# services/external/argo/helm/README.md "Internal Operations Notes" for the
# sizing methodology and the scale factor used for a given profile.
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
name: {{ staticstr + epoch_time }}
namespace: {{NAMESPACE}}
labels:
workflow: argo-scale-sw44057-retrystorm
job_name: "argo-scale-sw44057-retrystorm"
spec:
entrypoint: fanout
parallelism: {{RETRY_STORM_PARALLELISM | default(30)}}
podGC:
strategy: OnWorkflowSuccess
templates:
- name: fanout
dag:
tasks:
- name: flaky-task
template: flaky-task
arguments:
parameters:
- name: index
value: "{{ '{{item}}' }}"
withSequence:
count: "{{RETRY_STORM_TASK_COUNT | default(40)}}"
- name: flaky-task
inputs:
parameters:
- name: index
retryStrategy:
limit: "{{RETRY_STORM_LIMIT | default(20)}}"
retryPolicy: Always
backoff:
duration: "500ms"
factor: "1"
maxDuration: "5m"
container:
image: busybox:1.36
command: ["sh", "-c"]
# Deterministic persistent-failure pattern: a fixed fraction of task
# indices (controlled by RETRY_STORM_FAIL_MODULO) always fail and
# exhaust the full retryStrategy.limit, mirroring the incident's
# persistently-failing connector/LDAP tasks rather than transient
# flakiness. The remainder succeed immediately so the workflow still
# completes, allowing TTL/cleanup behavior to also be validated.
args:
- "if [ $(( {{ '{{inputs.parameters.index}}' }} % {{RETRY_STORM_FAIL_MODULO | default(40)}} )) -lt {{RETRY_STORM_FAIL_COUNT | default(34)}} ]; then echo task-{{ '{{inputs.parameters.index}}' }} failing permanently; exit 1; else echo task-{{ '{{inputs.parameters.index}}' }} succeeded; exit 0; fi"
resources:
requests:
cpu: 5m
memory: 8Mi
limits:
cpu: 20m
memory: 32Mi
```

### Logs from the workflow controller

```text
argo --server list
```

### Logs from in your workflow's wait container

```text
NA
```

Contributor guide

Open the contributing guide

Research direction

Start with persist/sqldb/offload_node_status_repo.go and workflow_server.go's ListWorkflows path, then compare the live workflow flow with the archived-list fixes in #12025, #13295, and #13601. Trace how the current page's workflow UIDs and filters are resolved before offloaded node status is loaded. Done means ListWorkflows no longer fetches unrelated namespace-wide blobs and avoids the reported MySQL max_allowed_packet failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, mysql
Domain
backend-api-design, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.