apache / apache/hamilton

Enable one to tag what should be sent to ray or dask via the respective GraphAdapter

Open
#613 0 comments 0 reactions 0 assignees View on GitHub
decorators Dynamic DAGs enhancement parallelism performance
Dominant language
Jupyter Notebook
Stars
2.6k
Forks
213
PR merge metrics
No merged PRs in 30d

Description

**Is your feature request related to a problem? Please describe.**
Sending to Ray or Dask every function can be costly from a serialization standpoint. We should be able to switch a mode that's the opposite -- we only annotate functions we want to farm out.

**Describe the solution you'd like**
```python
@tag(my_tag_key="my_tag_value")
def model_fitting_function() -> object:
# would be sent to Ray / Dask
return ...

def some_other_functions() -> ...:
# would not be sent to Ray / Dask
...

#--- in the driver
adapter = h_ray.RayGraphAdapter(..., tag_query=dict(my_tag_key="my_tag_value"))
...
```
We would need to benchmark this a little to see if this would be worthwhile to implement -- since again a lot depends on the shape of the graph, there might be better ways to specify what is or isn't good for "submitting" to Ray or Dask.

**Describe alternatives you've considered**
You use Ray or Dask outside Hamilton.
You use the Parallelizeable construct in Hamilton.

**Additional context**
There could be some edge cases to think through here.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.