Bogdanp / Bogdanp/dramatiq

Feature request: pass arguments automatically and individually in pipelines

Open
#335 0 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
5.3k
Forks
383
Avg merge
9h 41m
Merged PRs (30d)
2

Description

# Feature request

A form of pipeline that automatically passes its first input argument on to the next actor, along with any additional arguments as individual args.

```python
@dramatiq.actor()
def actor_1(list_of_numbers: List[Int]):
other_numbers = [5, 6, 7]
return other_numbers # -> Notice how we don't return 'list_of_numbers'

# -> Arguments are received in order, no need to access indices in a tuple
@dramatiq.actor()
def actor_2(list_of_numbers: List[Int], other_numbers: List[int]):
return list_of_numbers.extends(other_numbers)

pipeline([actor_1.message([1,2,3]), actor_2.message()]
```

If my understanding is correct, I would have to do the following to pass multiple args in dramatiq and maintain type hints:

```python
@dramatiq.actor()
def actor_1(list_of_numbers: List[Int]):
other_numbers = [5, 6, 7]
return list_of_numbers, other_numbers

@dramatiq.actor()
def actor_2(data): # -> No introspection or typing for function arguments :(
list_of_numbers: List[int] = data[0]
other_numbers: List[int] = data[1]

new_list = list_of_numbers.extends(other_numbers)
return list_of_numbers, new_list # -> Easy to confuse the order

pipeline([actor_1.message([1,2,3]), actor_2.message()]
```

# Benefits

* We don't have to keep passing 'list_of_numbers' manually through all the pipeline stages
* Feels a lot cleaner than manually accessing indices of a returned tuple
* Code introspection can tell us how many arguments an actor expects as well as the type of them.

# Issues

Breaking change since it fundamentally alters how arguments are passed between actors in a pipeline

# Ideal solution

Create a new composition type 'streamlined_pipeline' with this behavior or allow a keyword argument to be passed to 'pipeline' that modifies the behavior, i.e:

`pipeline(messages, pass_args=True)`

Contributor guide

Open the contributing guide

Research direction

Start at the existing pipeline(messages) entry point and inspect how actor return values become arguments for the next message. Define whether the behavior belongs in a new streamlined_pipeline type or an option on pipeline, then verify that subsequent actors receive the prior input and returned values as individual arguments without breaking existing behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.