DiamondLightSource / DiamondLightSource/blueapi

Triggering Argo Workflow pipelines

Open
#819 10 comments 0 reactions 0 assignees View on GitHub
external
Dominant language
Python
Stars
13
Forks
13
Avg merge
2d 11h
Merged PRs (30d)
25

Description

Been talking a lot with @olliesilvester today about how this is going to be accomplished, so wanted to get some words down.

Referencing and sometimes citing https://diamondlightsource.github.io/workflows/docs/

Ollie is working on enabling small in-process callbacks that are required for some plans. Because of the plan stub subscribe, this can be done from within a plan without needing to touch the RunEngine, but a way to make sure that the RunEngine unsubscribes again from the subscription id would be useful, else we risk gaining a growing overlapping set of callbacks with every run.

Because these callbacks are being run within the same python process as blueapi, they are limited by the resources given, which should remain minimal. When callbacks grow to require more resources or speed or repeatability, they should be extracted into the workflow system.

We therefore need to consider how to trigger those workflows. Blueapi already has one persistent callback to send documents to the message bus for the client and other services- the following assumes that this callback remains in place either to send messages to a bus or to insert documents into a document store.

In order to enable a complete experiment I believe we need:

- An API for workflows to consume data from [the message bus|a document store]
- An API for workflows to put data into [the message bus|a document store|the blueapi instance that spawned them]
- An API for plans to receive data that results from workflow runs and use them to adapt their future behaviour
- A way of triggering workflows: we should be able to decorate a plan and define 1 or many workflows that should be executed for [the plan|each run within the plan]

A simple plan that triggers an external workflow

```python
def my_plan(detectors: list[Readable], workflow_param: int):
@run_workflow(workflow_id="my_workflow", params={"my_param": workflow_param)
def inner_plan():
run_id = yield from open_run()
for _ in range(3):
yield from OurAPI.next_position_from_workflow(PositionType, "my_workflow", run_id)
yield from close_run()
```

A simple workflow, for which there is a copier template (or similar) to build a container and register it with the workflow engine with the name "my_workflow", such that the only thing required to create a new workflow is defining my_analysis or my_per_point_analysis or both (? Something entirely within this python file only)

```python
def my_analysis(foo: int, data) -> PositionType:
...

def my_per_point_analysis(foo: int, data) -> PositionType:
...

@click.command()
@click.argument('my_param')
def main(ctx: click.Context, my_param: int, run_id: uuid) -> None:
run = tiledAPI.run_from(run_id)
OurAPI.send(my_analysis(my_param, run.get_data()))
while run.has_more():
OurAPI.send(per_point_analysis(my_param, run.next_position())

```

## Acceptance Criteria
- All of the sub-issues on this issue have been resolved

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.