argoproj / argoproj/argo-workflows
Extend withParam field to support an inputs.artifacts.ARR values
- Dominant language
- Go
- Stars
- 17k
- Forks
- 3.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 138
Description
# Summary
Currently withParam only accepts a (json array)` inputs.parameters.ARR` as a value. e.g:
```
- - name: test-linux
template: cat-os-release
arguments:
parameters:
- name: image
value: "{{item.image}}"
- name: tag
value: "{{item.tag}}"
withParam: "{{inputs.parameters.os-list}}"
```
Allowing it to accept ` inputs.artifats.ARR` as its value will allow Argo to loop over very large arrays.
What change needs making?
copying here from slack thread:
> Hong Wang: Cool, we are on the same page. Technically, it is not as simple as it looks like. WithParam is actually done in the controller side which is running in argo namespaces, the controller can do it because the annotation is part of the pod spec, with the help of wait container, controller can access that info and expand withParam. The artifact is a different story, the main controller normally don't have the access to the secrets in other namespaces, so it won't be able to load the artifact file, and look into the JSON to expand it. Please do create an issue to track this. We are adding a feature called Argo-Agent might be able to solving this by running an additional agent pod inside user namespace to full-fill some controller responsibility. So, it could be promising to implement it in the end.
# Use Cases
When would you use this?
Currently, the only way to loop over a large array is via withSequence. it is cumbersome and hard to read and develop.
[link](https://github.com/argoproj/argo-workflows/blob/master/examples/handle-large-output-results.yaml)
This enhancement will allow the to store the big array (>256kb) in an Artifact and still be able to loop over it in the regular way.
---
**Message from the maintainers**:
Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.
Contributor guide
Research direction
Start with examples/handle-large-output-results.yaml, then trace the controller-side withParam expansion described in the issue and how artifact access differs across namespaces. Review the proposed Argo-Agent responsibility before deciding the implementation path. Done means workflows can use inputs.artifacts.ARR containing a large JSON array for withParam iteration.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- backend, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100