OpenFn / OpenFn/lightning

Faulty workflow.yaml can be saved from AI chat

Open
#3,307 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Elixir
Stars
296
Forks
86
Avg merge
1d 13h
Merged PRs (30d)
50

Description

@elias-ba recently pointed out that the workflow chat agent can send an invalid workflow.yaml to Lightning and we'll render it without question. This has lead to the generation of some broken workflows. (Namely, one with multiple "first nodes" all stemming directly from the trigger.)

Image

Saving a workflow like that would lead to all sorts of problems. We should validate the workflow.yaml that's being sent back from the AI assistant before rendering it. And if it's invalid, ideally, the workflow agent would get to see the error and try again.

On the workflow importer, we have a function that validates the proposed worklfow.yaml before attempting to render it on the canvas. Maybe we can use some of this?

Version number v2.13.1

I have reproduced this locally on main:

  • Yes
  • No

To Reproduce Steps to reproduce the behavior:

  1. Go to the workflow chat
  2. Ask it to generate a job with lots of different "initial nodes" stemming directly from the cron trigger.

--

P.S., @elias-ba , are you certain we don't have an issue for this? I couldn't find one either, but I swear that @josephjclark , @hanna-paasivirta, or @stuartc was talking about an Apollo-side call to a validation service and I'd be surprised if there wasn't an issue somewhere for checking the generated workflow.yaml before sending it back to lightning—possibly even with an externally available "workflow validator API" or something like that.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the workflow chat entry point and the workflow importer validation function mentioned in the issue, then trace where the generated workflow.yaml is rendered on the canvas. Done means invalid workflows are rejected before rendering and the validation error is made available to the workflow agent for another attempt.

Written by the indexing model from the issue text.

Assessment

Tech stack
elixir
Domain
ai, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.