apache / apache/hamilton

Add ability to mark function outputs as unserializable

Open
#743 2 comments 0 reactions 0 assignees View on GitHub
decorators Dynamic DAGs enhancement parallelism
Dominant language
Jupyter Notebook
Stars
2.6k
Forks
213
PR merge metrics
No merged PRs in 30d

Description

**Is your feature request related to a problem? Please describe.**
In Parallel & Collect blocks, there could be a process boundary. Inputs that are invariant to each chunk of the block need to be serialized and deserialized -- but for things like clients (e.g. openai, db client) those objects aren't serializable and things break.

**Describe the solution you'd like**
```python

@unserializable
def client() -> object:
"""Some client, e.g. openai, etc"""
return ...
```
Then Hamilton will know that for each parallel block, we rerun this function, versus trying to run it once and trying to serialize and deserialize it.

**Describe alternatives you've considered**
You can get around this by making the `client()` function depend on some part of the parallel block (`client(some_dep: object)`) thereby forcing it to be run each block.

**Additional context**
This would be a quality of life improvement.

Contributor guide

Open the contributing guide

Research direction

Start by tracing how Parallel & Collect blocks serialize invariant inputs and how function metadata or decorators are represented. Define how an @unserializable function is identified and rerun per parallel block, then verify the behavior with a client-like function that cannot be serialized; the issue does not name specific files or tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.