kestra-io / kestra-io/plugin-graalvm

Control columns order for FileTransform tasks

Open
#91 0 comments 0 reactions 0 assignees View on GitHub
area/plugin good first issue
Dominant language
Java
Stars
0
Forks
1
Avg merge
5h 23m
Merged PRs (30d)
4

Description

### Feature description

Sometimes it is useful to define a strict order of column outputted by FileTransform task (jython in my case). For example, when I want to send data to Batch task for storage or when I need to produce a specific report (CSV, etc.) Currently, all I can do is try to adapt following tasks inputs which is fragile 'cause I can not predict columns order of previous task beforehand and this behaviour is not documented/guaranteed. E.g. if I add some new column to FileTransform the whole order can change.

It would be really nice to be able to specify column order explicitly. Either via some additional task properties or maybe using some intermediate new task that interacts with ION data.

Context: https://kestra-io.slack.com/archives/C03FQKXRK3K/p1720467171927799

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the FileTransform task and reviewing how its task properties and output columns are handled. Decide how an explicit column-order option should be represented, then verify that the configured order is preserved for downstream Batch storage and CSV-style reports and that the behavior is documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.