apache / apache/hamilton

Data loaders should be able to inject tags to the graph

Open
#611 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Jupyter Notebook
Stars
2.6k
Forks
213
PR merge metrics
No merged PRs in 30d

Description

**Is your feature request related to a problem? Please describe.**
I'm a central platform team. I own some data adapters. I want to be able to inject some tags into the graph on the nodes.

**Describe the solution you'd like**
Data loaders can be used in many contexts. So I think the only way to get some tags is after instantiation, and a function is called.

Something like
```python
@dataclasses.dataclass
class JSONDataLoader(DataLoader):
path: str

@classmethod
def applicable_types(cls) -> Collection[Type]:
return [dict]

def load_data(self, type_: Type) -> Tuple[dict, Dict[str, Any]]:
with open(self.path, "r") as f:
return json.load(f), get_file_metadata(self.path)

@classmethod
def name(cls) -> str:
return "json"

def tags(self) -> dict: # <------------- add something like this
return {"some_special_tag": self.path, ...}
```

**Describe alternatives you've considered**
It's not critical, but something to think about.

**Additional context**
If one was to build more capabilities using tag metadata, then we'd want this here.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.