apache / apache/hamilton

Enable configuration/tags to turn off data introspection in SDK

Open
#921 0 comments 0 reactions 0 assignees View on GitHub
enhancement SDK UI
Dominant language
Jupyter Notebook
Stars
2.6k
Forks
213
PR merge metrics
No merged PRs in 30d

Description

**Is your feature request related to a problem? Please describe.**
Currently the SDK introspects all outputs. Sometimes, especially in the case of Parallel, this could be a really long list of things; it'd be nice to turn it off selectively.

**Describe the solution you'd like**
Option 1: inspect tag to skip
```python
@tag(sdk=["skip.introspection"])
def my_func(...) -> ...:
```

Option 2: skip via configuration
```python
sdk_config = {"my_func": ["skip.introspection"]}
HamiltonTracker(..., tracker_config=sdk_config) # or some config file
```

---
We'd then want these to both output some type of result that the UI could then render.
Minimally based on the type returned it could be something basic about it -- so when you got to the UI you can see that this was skipped.

**Describe alternatives you've considered**
N/A

**Additional context**
This is an optimization.

Functionality to hide sensitive information from being logged could follow a similar path -- but likely a more explicit decorator in the code would be best in this situation.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.