argoproj / argoproj/argo-workflows
Passing --loglevel error to workflow-controller does not change the log level
- Dominant language
- Go
- Stars
- 17k
- Forks
- 3.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 138
Description
### Pre-requisites
- [x] I have double-checked my configuration
- [x] I have tested with the `:latest` image tag (i.e. `quay.io/argoproj/workflow-controller:latest`) and can confirm the issue still exists on `:latest`. If not, I have explained why, **in detail**, in my description below.
- [x] I have searched existing issues and could not find a match for this bug
- [ ] I'd like to contribute the fix myself (see [contributing guide](https://github.com/argoproj/argo-workflows/blob/main/docs/CONTRIBUTING.md))
### What happened? What did you expect to happen?
I deployed argowf v3.5.7 and ran into high log rates due its default info level. I attempted to reduce this by passing `--loglevel error` to the application, yet I am still seeing info logs from the workflow-controller pods. I've also passed `--gloglevel 0`, `--gloglevel 5` and `--gloglevel -5` to see if those would help. None did. Passing `--loglevel error` to the argo server pods did work.
There seems to be an issue passing this log level to the actual logging system in the workflow-controller.
Full Deployment yaml:
```yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: workflow-controller
namespace: argowf
spec:
replicas: 2
selector:
matchLabels:
app: workflow-controller
template:
metadata:
labels:
app: workflow-controller
spec:
containers:
- args:
- --loglevel
- error
command:
- workflow-controller
env:
- name: LEADER_ELECTION_IDENTITY
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: TRANSIENT_ERROR_PATTERN
value: .*context deadline exceeded.*
image: quay.io/argoproj/workflow-controller:v3.6.7
livenessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 6060
initialDelaySeconds: 90
periodSeconds: 60
timeoutSeconds: 30
name: workflow-controller
ports:
- containerPort: 9090
name: metrics
- containerPort: 6060
securityContext:
allowPrivilegeEscalation: false
appArmorProfile:
type: RuntimeDefault
capabilities:
drop:
- ALL
privileged: false
readOnlyRootFilesystem: true
runAsGroup: 1000
runAsNonRoot: true
runAsUser: 1000
seccomProfile:
type: RuntimeDefault
nodeSelector:
kubernetes.io/os: linux
priorityClassName: workflow-controller
securityContext:
runAsNonRoot: true
serviceAccountName: argo
```
### Version(s)
v3.5.7
### Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflow that uses private images.
```YAML
workflow itself is not necessary to debug this.
```
### Logs from the workflow controller
```text
time="2025-05-01T11:15:40Z" level=info msg="index config" indexWorkflowSemaphoreKeys=true
time="2025-05-01T11:15:40Z" level=info msg="cron config" cronSyncPeriod=10s
time="2025-05-01T11:15:40Z" level=info msg="Memoization caches will be garbage-collected if they have not been hit after" gcAfterNotHitDuration=30s
I0501 11:15:40.763025 1 leaderelection.go:257] attempting to acquire leader lease argowf/workflow-controller...
I0501 11:15:56.786136 1 leaderelection.go:271] successfully acquired lease argowf/workflow-controller
W0501 11:15:57.220430 1 shared_informer.go:463] The sharedIndexInformer has started, run more than once is not allowed
W0501 11:16:00.367619 1 warnings.go:70] metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must be no more than 63 characters]
time="2025-05-01T11:16:00Z" level=info msg="add pod event" pod=sessieworkflow-javacronworkflow-1746098160-startsessie-2209688735
time="2025-05-01T11:16:00Z" level=info msg="insignificant pod change" key=dat/sessieworkflow-javacronworkflow-1746098160-startsessie-2209688735
time="2025-05-01T11:16:00Z" level=info msg="update pod event" pod=sessieworkflow-javacronworkflow-1746098160-startsessie-2209688735
time="2025-05-01T11:16:00Z" level=info msg="update pod event" pod=sessieworkflow-javacronworkflow-1746098160-startsessie-2209688735
time="2025-05-01T11:16:00Z" level=info msg="add pod event" pod=parsemsdwworkflow-parserworkflow-1746098160-parsemsdwfilestoworkflow-747416414
W0501 11:16:00.510610 1 warnings.go:70] metadata.annotations[container.apparmor.security.beta.kubernetes.io/main]: deprecated since v1.30; use the "appArmorProfile" field instead
W0501 11:16:00.510638 1 warnings.go:70] metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must be no more than 63 characters]
time="2025-05-01T11:16:00Z" level=info msg="update pod event" pod=parsemsdwworkflow-parserworkflow-1746098160-parsemsdwfilestoworkflow-747416414
```
### Logs from in your workflow's wait container
```text
-
```
Contributor guide
Assessment
This issue has not been assessed yet.