argoproj / argoproj/argo-workflows
Setting `ALWAYS_OFFLOAD_NODE_STATUS` without setting `nodeStatusOffLoad` results in workflow errors
- Dominant language
- Go
- Stars
- 17k
- Forks
- 3.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 138
Description
### Pre-requisites
- [X] I have double-checked my configuration
- [X] I can confirm the issue exists when I tested with `:latest`
- [X] I have searched existing issues and could not find a match for this bug
- [ ] I'd like to contribute the fix myself (see [contributing guide](https://github.com/argoproj/argo-workflows/blob/main/docs/CONTRIBUTING.md))
### What happened/what did you expect to happen?
I was in the process of enabling node status offloading. As part of this I set the `ALWAYS_OFFLOAD_NODE_STATUS` environment variable and missed making the `nodeStatusOffload` property in the configmap to true.
This resulted in the workflows failing with the error: `offload node status is not supported` (cli) and `Workflow operation error` (ui).
While this is definitely a configuration issue, it would be nice to have the controller ignore the `ALWAYS_OFFLOAD_NODE_STATUS` flag if the `nodeStatusOffload` setting is set to false so that users don't have to disable both if they need to disable node status offloading.
### Version
latest
### Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.
```YAML
# This is an issue with any workflow, so pasting the default workflow here.
metadata:
name: fantastic-python
labels:
example: 'true'
spec:
arguments:
parameters:
- name: message
value: hello argo
entrypoint: argosay
templates:
- name: argosay
inputs:
parameters:
- name: message
value: '{{workflow.parameters.message}}'
container:
name: main
image: 'argoproj/argosay:v2'
command:
- /argosay
args:
- echo
- '{{inputs.parameters.message}}'
ttlStrategy:
secondsAfterCompletion: 300
podGC:
strategy: OnPodCompletion
```
### Logs from the workflow controller
```text
Type Reason Age From Message
---- ------ ---- ---- -------
Normal WorkflowRunning 65s workflow-controller Workflow Running
Warning WorkflowFailed 65s workflow-controller offload node status is not supported
```
### Logs from in your workflow's wait container
```text
This was not done :(
```
Contributor guide
Research direction
Start by tracing how the controller reads the ALWAYS_OFFLOAD_NODE_STATUS environment variable and the nodeStatusOffload ConfigMap setting. Reproduce the issue with the provided workflow and mismatched settings, then verify that disabling node status offloading allows the workflow to complete without the reported error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- backend, devops
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100