apache / apache/hamilton

enable free-form attributes on nodes

Open
#1,129 3 comments 0 reactions 0 assignees View on GitHub
decorators
Dominant language
Jupyter Notebook
Stars
2.6k
Forks
213
PR merge metrics
No merged PRs in 30d

Description

In some scenarios, for purposes unrelated to DAG execution, the user wants to attach free-form information to nodes. This feature request provides some desirable usage examples.

Currently, the tags mechanism _could_ be used to attach immutable attributes to the node. Besides being immutable, the attributes must be converted to and from objects to strings, which is sub-optimal.

Free-form attributes are much like tags, but free-form. Unlike tags, Hamilton should never be tasked to interpret attributes, just to accumulate them along node expansion paths and eventually store them on a HamiltonNode.

Example 1
Common attributes for expanded nodes
```python
a_value = {}
b_value = dict(abc=[])
common_attrs = dict(a=a_value, b=b_value)
@parameterize(
o1=dict(x=source('o1_x_input')),
o2=dict(x=source('o2_x_input')),
)
@attr(common_attrs)
def fun(x: dict, i: int) -> dict:
return x
```
Here I expect HamiltonNodes o1.attributes and o2.attributes to be {**common_attrs} each.

Example 2
Provide distinct attributes for each expanded node

```python
o1_attrs = dict(name='o1')
o2_attrs = dict(name='o2')
@parameterize(
o1=dict(x=source('o1_x_input'), None=value(o1_attrs)),
o2=dict(x=source('o2_x_input'), None=value(o2_attrs)),
)
@attr(common_attrs)
def fun(x: dict, i: int) -> dict:
return x
```

Note the None key in the parameterize args.
Here I expect HamiltonNodes o1.attributes == {**common_attrs, **o1_attrs}

Example 3

Use attributes to capture the function.
I should be able to roll up my own decorator, say `capture_func_name` and use it like this

```python
@capture_func_name
def fun(x: dict, i: int) -> dict:
return x
```

which should be equivalent to

```python
def fun(x: dict, i: int) -> dict:
return x

fun = attr(dict(func=fun))(fun)
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.