OpenFn / OpenFn/lightning

Identifiy and fix broken cron workflows

Open
#4,624 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Elixir
Stars
296
Forks
86
Avg merge
1d 13h
Merged PRs (30d)
50

Description

We have a problem where empty cron workflows recurse forever - see https://github.com/OpenFn/kit/issues/1367

I have put a bandaid on the problem for new workflows. But existing ones may still give us problems.

This issue is two-fold:

  1. identify any existing workflows on prod and staging which may trigger this behaviour
  2. roll out a fix

Identification

We need to work out which workflows are affected.

These things must be true:

  • The workflow must be on a cron trigger
  • The workflow must have empty bodies (empty is not nil, it could just mean a comment)
  • The workflow must have multiple leaf nodes. Probably not enough to just have true/false branches - it's gotta have true/true edges

I don't know which of these we can use to build a query - some of them are quite hard heuristics

What might help is: the worker will detect runs with multiple leaf nodes. If a run does NOT have multiple leaves, it'll include final_dataclip_id. So maybe: identify cron runs which did not return final_dataclip_id in the final payload - those are the ones we need to fix.

For the record, this affects at least one workflow on prod: https://app.openfn.org/projects/57862287-23e6-4650-8d79-e1dd88b24b1c/w/0f1d0b6e-8426-4023-959d-a89ba824f3c0

Fix

Consider the following:

  • Edit the last dataclip - the source for the next cron - to be an empty object {}
  • Delete or edit prior dataclips, which could be taking up a lot of storage and providing no value
  • Disabling the cron trigger (bear in mind that if a user switches it back on, they'll just provoke the problem again)

When deciding how aggressively to fix this, we need to consider if we're looking at demo code or a real project.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the worker behavior described in OpenFn/kit#1367 and inspect cron runs whose final payload lacks final_dataclip_id. Identify affected workflows on production and staging using the listed trigger, empty-body, and multiple-leaf criteria; done means affected workflows are found and their dataclips or cron triggers are safely fixed.

Written by the indexing model from the issue text.

Assessment

Tech stack
elixir
Domain
backend, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.