argoproj / argoproj/argo-workflows

Setting `ALWAYS_OFFLOAD_NODE_STATUS` without setting `nodeStatusOffLoad` results in workflow errors

Open
#12,563 3 comments 0 reactions 0 assignees View on GitHub
area/controller P3
Dominant language
Go
Stars
17k
Forks
3.7k
Avg merge
1d 20h
Merged PRs (30d)
138

Description

### Pre-requisites

- [X] I have double-checked my configuration
- [X] I can confirm the issue exists when I tested with `:latest`
- [X] I have searched existing issues and could not find a match for this bug
- [ ] I'd like to contribute the fix myself (see [contributing guide](https://github.com/argoproj/argo-workflows/blob/main/docs/CONTRIBUTING.md))

### What happened/what did you expect to happen?

I was in the process of enabling node status offloading. As part of this I set the `ALWAYS_OFFLOAD_NODE_STATUS` environment variable and missed making the `nodeStatusOffload` property in the configmap to true.

This resulted in the workflows failing with the error: `offload node status is not supported` (cli) and `Workflow operation error` (ui).

While this is definitely a configuration issue, it would be nice to have the controller ignore the `ALWAYS_OFFLOAD_NODE_STATUS` flag if the `nodeStatusOffload` setting is set to false so that users don't have to disable both if they need to disable node status offloading.

### Version

latest

### Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

```YAML
# This is an issue with any workflow, so pasting the default workflow here.
metadata:
name: fantastic-python
labels:
example: 'true'
spec:
arguments:
parameters:
- name: message
value: hello argo
entrypoint: argosay
templates:
- name: argosay
inputs:
parameters:
- name: message
value: '{{workflow.parameters.message}}'
container:
name: main
image: 'argoproj/argosay:v2'
command:
- /argosay
args:
- echo
- '{{inputs.parameters.message}}'
ttlStrategy:
secondsAfterCompletion: 300
podGC:
strategy: OnPodCompletion
```

### Logs from the workflow controller

```text
Type Reason Age From Message
---- ------ ---- ---- -------
Normal WorkflowRunning 65s workflow-controller Workflow Running
Warning WorkflowFailed 65s workflow-controller offload node status is not supported
```

### Logs from in your workflow's wait container

```text
This was not done :(
```

Contributor guide

Open the contributing guide

Research direction

Start by tracing how the controller reads the ALWAYS_OFFLOAD_NODE_STATUS environment variable and the nodeStatusOffload ConfigMap setting. Reproduce the issue with the provided workflow and mismatched settings, then verify that disabling node status offloading allows the workflow to complete without the reported error.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
backend, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.