apache / apache/hamilton

Add resolved_kwargs to data_saver and data_loader tags

Open
#1,136 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
2.6k
Forks
213
PR merge metrics
No merged PRs in 30d

Description

**Is your feature request related to a problem? Please describe.**
When I have a built dataflow I would like to be able to see which paths are entered in @load_from and @save_to.

**Describe the solution you'd like**
After executing the dataflow I can see the paths in the results, but I'd like to be able to see them without executing the dataflow.

Some metadata is already being written to tags: https://github.com/DAGWorks-Inc/hamilton/blob/main/hamilton/function_modifiers/adapters.py#L578

I tested adding the following line there:
```python
"hamilton.data_saver.kwargs": resolved_kwargs,
```
Then I tried running `examples/parallelism/star_counting/run.py` with the dr.execute statement replaced by:
```python
node = next(
node for node in dr.list_available_variables() if node.name == "save.unique_stargazers"
)
print(node.as_dict()["tags"])
```
This gives the output I was hoping for:
```python
{'hamilton.data_saver': True, 'hamilton.data_saver.sink': 'csv', 'hamilton.data_saver.classname': 'PandasCSVWriter', 'hamilton.data_saver.kwargs': {'path': 'unique_stargazers.csv'}}
```

**Describe alternatives you've considered**
Maybe a custom DataLoader and DataSaver that store the arguments they were initiated with?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.