OpenFn / OpenFn/lightning

Make work order-specific queues blocking (Run Concurrency)

Open
#2,907 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

needs detail
Dominant language
Elixir
Stars
296
Forks
86
Avg merge
1d 13h
Merged PRs (30d)
50

Description

Make work order-specific queues blocking (Enforce order/sequence);

Via the workflow settings panel, we want to enable users to the able to:

  1. Specify that for a given workflow, work order 3 should not be processed until work order 2 reaches state = success || ignored
  2. Allow users to specify that for a given workflow, work order 7 should not be processed until work order 4 reaches state = success || ignored if both meet the same key criterion. (This feature is required by Kafka trigger as well to enforce sequence in the how messages are processed by the consumer group)
Select from runs
	where state = available
	and no other workorders for this workflow with dataclip.body.arbitrary_key is yet
        to have state  = [success|skipped]
  1. Allow users to specify concurrency per-arbitrary-dataclip-key-per-workflow.
So if we have 10 runs enqueued (3 with the same arbitrary-data clip-value: dataclip.body.xyx = “myguy!”) 
then we’d do 8-at-a-time, but keep the 2 other dataclip.body.xyx = “myguy!” containing runs BLOCKED
 until the first was done.
  1. Allow users to mark work orders in the ignored state
update work order set state = ‘ignored’ where wo.id = this_wo_id
  1. Allow users to clear (set to ignore, never process) pending runs/workorders in their queue
update runs set state = ‘skipped’ where runs.state = ‘available’
update work order state if the skipped run is only run for that workorder
  1. Maybe make it easier to find work orders to retry that include skipped runs?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the workflow settings panel, the run/work-order queue behavior, and the Kafka trigger or consumer-group entry point named in the issue. The issue names no files or tests; done would require agreeing on and implementing the six listed ordering, concurrency, ignored-state, clearing, and retry behaviors.

Written by the indexing model from the issue text.

Assessment

Tech stack
elixir
Domain
backend, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.