argoproj / argoproj/argo-workflows

Data Sourcing and Transformations: add jsonpath as possible transform

Open
#8,432 0 comments 8 reactions 0 assignees View on GitHub
area/templates/data
Dominant language
Go
Stars
17k
Forks
3.7k
Avg merge
1d 20h
Merged PRs (30d)
138

Description

# Summary
Add the ability to use jsonpath as part of the data transform step

# Use Cases
1. You input some structured json data either from an external source, or from a previous step's artifact in the workflow.
2. You want to filter, map, or select a specific value from the json so you can pass it to a job downstream, without having to write a bash script.
I looked briefly at expr and the existing example for data sourceing, and I couldn't find a way to achieve my use case (specifically selecting a single value from an artifact json generated in a previous step).
---

**Message from the maintainers**:

Love this enhancement proposal? Give it a 👍. We prioritize the proposals with the most 👍.

Contributor guide

Open the contributing guide

Research direction

Start by reading the existing data-sourcing example and the current expr-based transform support to understand how workflow artifacts are transformed. The work is done when structured JSON can be filtered, mapped, or queried with jsonpath and the selected result can be passed to a downstream job without a bash script.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.