Enable one to tag what should be sent to ray or dask via the respective GraphAdapter
- Dominant language
- Jupyter Notebook
- Stars
- 2.6k
- Forks
- 213
- PR merge metrics
- No merged PRs in 30d
Description
**Is your feature request related to a problem? Please describe.**
Sending to Ray or Dask every function can be costly from a serialization standpoint. We should be able to switch a mode that's the opposite -- we only annotate functions we want to farm out.
**Describe the solution you'd like**
```python
@tag(my_tag_key="my_tag_value")
def model_fitting_function() -> object:
# would be sent to Ray / Dask
return ...
def some_other_functions() -> ...:
# would not be sent to Ray / Dask
...
#--- in the driver
adapter = h_ray.RayGraphAdapter(..., tag_query=dict(my_tag_key="my_tag_value"))
...
```
We would need to benchmark this a little to see if this would be worthwhile to implement -- since again a lot depends on the shape of the graph, there might be better ways to specify what is or isn't good for "submitting" to Ray or Dask.
**Describe alternatives you've considered**
You use Ray or Dask outside Hamilton.
You use the Parallelizeable construct in Hamilton.
**Additional context**
There could be some edge cases to think through here.
Contributor guide
Assessment
This issue has not been assessed yet.