DiamondLightSource / DiamondLightSource/blueapi
Triggering Argo Workflow pipelines
- Dominant language
- Python
- Stars
- 13
- Forks
- 13
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 25
Description
Been talking a lot with @olliesilvester today about how this is going to be accomplished, so wanted to get some words down.
Referencing and sometimes citing https://diamondlightsource.github.io/workflows/docs/
Ollie is working on enabling small in-process callbacks that are required for some plans. Because of the plan stub subscribe, this can be done from within a plan without needing to touch the RunEngine, but a way to make sure that the RunEngine unsubscribes again from the subscription id would be useful, else we risk gaining a growing overlapping set of callbacks with every run.
Because these callbacks are being run within the same python process as blueapi, they are limited by the resources given, which should remain minimal. When callbacks grow to require more resources or speed or repeatability, they should be extracted into the workflow system.
We therefore need to consider how to trigger those workflows. Blueapi already has one persistent callback to send documents to the message bus for the client and other services- the following assumes that this callback remains in place either to send messages to a bus or to insert documents into a document store.
In order to enable a complete experiment I believe we need:
- An API for workflows to consume data from [the message bus|a document store]
- An API for workflows to put data into [the message bus|a document store|the blueapi instance that spawned them]
- An API for plans to receive data that results from workflow runs and use them to adapt their future behaviour
- A way of triggering workflows: we should be able to decorate a plan and define 1 or many workflows that should be executed for [the plan|each run within the plan]
A simple plan that triggers an external workflow
```python
def my_plan(detectors: list[Readable], workflow_param: int):
@run_workflow(workflow_id="my_workflow", params={"my_param": workflow_param)
def inner_plan():
run_id = yield from open_run()
for _ in range(3):
yield from OurAPI.next_position_from_workflow(PositionType, "my_workflow", run_id)
yield from close_run()
```
A simple workflow, for which there is a copier template (or similar) to build a container and register it with the workflow engine with the name "my_workflow", such that the only thing required to create a new workflow is defining my_analysis or my_per_point_analysis or both (? Something entirely within this python file only)
```python
def my_analysis(foo: int, data) -> PositionType:
...
def my_per_point_analysis(foo: int, data) -> PositionType:
...
@click.command()
@click.argument('my_param')
def main(ctx: click.Context, my_param: int, run_id: uuid) -> None:
run = tiledAPI.run_from(run_id)
OurAPI.send(my_analysis(my_param, run.get_data()))
while run.has_more():
OurAPI.send(per_point_analysis(my_param, run.next_position())
```
## Acceptance Criteria
- All of the sub-issues on this issue have been resolved
Contributor guide
Assessment
This issue has not been assessed yet.