Add ability to mark function outputs as unserializable
- Dominant language
- Jupyter Notebook
- Stars
- 2.6k
- Forks
- 213
- PR merge metrics
- No merged PRs in 30d
Description
**Is your feature request related to a problem? Please describe.**
In Parallel & Collect blocks, there could be a process boundary. Inputs that are invariant to each chunk of the block need to be serialized and deserialized -- but for things like clients (e.g. openai, db client) those objects aren't serializable and things break.
**Describe the solution you'd like**
```python
@unserializable
def client() -> object:
"""Some client, e.g. openai, etc"""
return ...
```
Then Hamilton will know that for each parallel block, we rerun this function, versus trying to run it once and trying to serialize and deserialize it.
**Describe alternatives you've considered**
You can get around this by making the `client()` function depend on some part of the parallel block (`client(some_dep: object)`) thereby forcing it to be run each block.
**Additional context**
This would be a quality of life improvement.
Contributor guide
Research direction
Start by tracing how Parallel & Collect blocks serialize invariant inputs and how function metadata or decorators are represented. Define how an @unserializable function is identified and rerun per parallel block, then verify the behavior with a client-like function that cannot be serialized; the issue does not name specific files or tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100