StackStorm / StackStorm/st2

Allow concurrency.attr policies to use set arithmetic over a set of items

Open
#3,282 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

feature proposal
Dominant language
Python
Stars
6.5k
Forks
787
PR merge metrics
No merged PRs in 30d

Description

Feature proposal

I would like to extend the concurrency.attr policy to support simple collections. Set arithmetic could be used to allow for 2 actions to collide if there is a set intersection, causing the subsequent action to be delayed.

I would like to be able to express this policy in a manner similar to:

---
  name: prevent_overlapping_objects.concurrency
  description: Limits the concurrent executions for the workflow
  enabled: true
  resource_ref: my_pack.update_remote_object
  policy_type: action.concurrency.attr
  parameters:
    threshold: 1
    attributes:
      - object_uuids

Where object_uuids is a list of strings: ['268b7666-4e02-4672-bf0d-820121480111', '619427ec-2b19-4b51-9dde-af2697a4dfeb']

Now suppose I have an action that operations on multiple objects:


---
name: sync_remote_objects
runner_type: python-script
description: Updates remote objects
enabled: true
entry_point: sync_remote_objects.py
parameters:
  object_uuids:
    type: object
    description: List of remote object UUIDs.
    required: true
    default: [] 

And suppose I have 2 event driven workflows (A, B) which both result in calling sync_remote_objects with the object_uuids parameters. The goal is to identify that while workflow A is running the action, if workflow B attempts to also call the same action and if there is a set intersection on the concurrency.attr parameter (object_uuids here) , then the action instance in workflow B is delayed.

It seems that at a minimum a change would be required to the filtering and query capabilities in concurrency_by_attr.py and a new policy type would probably need to be defined. Or perhaps this should be a separate module with a new set of capabilities?

Motivation

I'm working on a project using stackstorm which manage aspects of configuration between several devices on the network.

Without getting into the details, I'll just say that we have been designing around this concurrency limitation for some time, and the impact on the design changes is not ideal. As remote object relationships being managed becomes more complex, we are being forced to consolidate more logic into single-threaded long running actions to maintain consistency, at the cost of performance.

I'll try to come up with a draft PR, but I'm interested in the team's feedback on this idea.

Thanks.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with concurrency_by_attr.py and the action-concurrency policy documentation linked in the issue. Trace how attributes are filtered and queried, then determine the policy entry point needed for collection values. Done should be defined by tests or examples showing that overlapping object_uuids delay the subsequent action while disjoint sets can run concurrently.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, devops
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.