argoproj / argoproj/argo-workflows
Regression in v3.7.11: jsonpath() on withSequence count=0 now errors instead of skipping (#15442)
- Dominant language
- Go
- Stars
- 17k
- Forks
- 3.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 138
Description
### Pre-requisites
- [x] I have double-checked my configuration
- [x] I have tested with the `:latest` image tag (i.e. `quay.io/argoproj/workflow-controller:latest`) and can confirm the issue still exists on `:latest`. If not, I have explained why, **in detail**, in my description below.
- [x] I have searched existing issues and could not find a match for this bug
- [ ] I'd like to contribute the fix myself (see [contributing guide](https://github.com/argoproj/argo-workflows/blob/main/docs/CONTRIBUTING.md))
### What happened? What did you expect to happen?
Workflows that ran successfully on **v3.7.10** now transition to `Error` on **v3.7.11+** with identical input, when a template uses `withSequence` with a dynamic `count` that can be `0` and downstream arguments use `{{=jsonpath(inputs.parameters.X, '$.start')}}`.
**Expected:** since the v3.7.11 and v3.7.12 release notes and [`docs/upgrading.md`](https://github.com/argoproj/argo-workflows/blob/main/docs/upgrading.md) do not flag any breaking changes, the previous behaviour should replicate
on the new versions. On v3.7.10 the task with `withSequence` count `0` is skipped cleanly, and the `jsonpath()` expression — which cannot be meaningfully evaluated because the sequence has no iterations — does not fail the task.
**Observed (v3.7.11+)**: the task transitions to `Error` with
```
failed to evaluate expression: invalid character '{' looking for beginning of object key string (1:1)
| jsonpath(inputs.parameters.X, '$.start')
| ^
```
We have traced the change in behaviour to PR [#15442](https://github.com/argoproj/argo-workflows/pull/15442) (cherry-pick [#15657](https://github.com/argoproj/argo-workflows/pull/15657) for release-3.7), shipped in v3.7.11. The new `expressionReplaceStrict` path resolves `{{=jsonpath(...)}}` during argument resolution before the sequence-length check. Because the identifier `inputs.parameters.X` is present in scope (as an unresolved placeholder string, not valid JSON), `allowUnresolved` is set to `false` and `jsonpath()`'s runtime panic on invalid JSON is no longer tolerated.
We rely on the release notes and [`docs/upgrading.md`](https://github.com/argoproj/argo-workflows/blob/main/docs/upgrading.md) to surface breaking changes and changes in behavior between releases to assess how these changes would affect the Argo workflows users in our internal platform.
But neither of these docs carries any mention of this change — the commit is listed as a plain `fix:` bullet, and `docs/upgrading.md` has no entry for v3.7.11 or v3.7.12. Because it was categorized as a fix, we realized about the change in behavior only when our internal users reported that their workflows which were running fine before the new upgrade was rolled out had started to fail.
Guidance on how users are expected to become aware of breaking changes that ship under a non-breaking classification (e.g. `fix:`) would be very helpful — that is the real ask behind this report.
## Version(s)
- Last known-good: **v3.7.10**
- First affected: **v3.7.11** (cherry-pick PR #15657, commit `477442702`)
- Also affected: **v3.7.12**, **v3.7.13** (current release-3.7 head)
### Version(s)
v3.7.11, v3.7.12
### Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.
```YAML
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: withsequence-zero-jsonpath-
spec:
entrypoint: main
templates:
- name: main
dag:
tasks:
- name: partition
template: partition
- name: convert
dependencies: [partition]
template: convert
withSequence:
count: "{{tasks.partition.outputs.parameters.total_chunks}}"
arguments:
parameters:
- name: start
value: "{{=jsonpath(inputs.parameters.time_range, '$.start')}}"
- name: index
value: "{{item}}"
- name: partition
outputs:
parameters:
- name: total_chunks
value: "0"
- name: time_range
value: '{"start":"2025-01-01","end":"2025-01-02"}'
container:
image: alpine:3
command: [sh, -c, "true"]
- name: convert
inputs:
parameters:
- name: start
- name: index
container:
image: alpine:3
command: [sh, -c, "echo start=$0 index=$1", "{{inputs.parameters.start}}", "{{inputs.parameters.index}}"]
```
On v3.7.10 this workflow succeeds (the `convert` task is marked `Skipped, empty params`). On v3.7.11+ it fails with the error.
### Logs from the workflow controller
```text
Key line observed on v3.7.12:
time="..." level=error msg="failed to evaluate expression: invalid character '{' looking for beginning of object key string (1:1)\n | jsonpath(inputs.parameters.time_range, '$.start')\n | ^" namespace=argo workflow=withsequence-zero-jsonpath-
```
### Logs from in your workflow's wait container
```text
No wait container is created for the failing task — the task never starts because argument resolution fails before pod creation.
```
Contributor guide
Research direction
Read PR #15442 and cherry-pick #15657, then trace the expressionReplaceStrict argument-resolution path described in the report. Reproduce the minimal workflow on v3.7.10 and v3.7.11+, and review docs/upgrading.md; done means the zero-count task skips without evaluating the unresolved jsonpath expression and the behavior-change guidance is addressed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- backend, devops
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100