Combined `extract` decorator
- Dominant language
- Jupyter Notebook
- Stars
- 2.6k
- Forks
- 213
- PR merge metrics
- No merged PRs in 30d
Description
**Is your feature request related to a problem? Please describe.**
We have two separate codepaths for `extract_fields` and `extract_columns`.
Furthermore, `extract_fields` requires one to write the type in the extract, but we should have that information in the fn output.
I think we can simplify the code quite a bit.
**Describe the solution you'd like**
A single `extract` (name TBD) decorator that does the following:
```python
@extract("foo", "bar")
def foo_bar(foo: int, bar: int) -> Dict[str, int]:
return {"foo": foo, "bar": bar}
@extract("foo", "bar")
def foo_bar(foo: pd.Series, bar: pd.Series) -> pd.DataFrame:
return pd.DataFrame({"foo": foo, "bar": bar})
```
Then it can register "extensions" to allow for us to processing different extractions. E.G. different DF types, etc...
If its a `Dict[str, foo]`, we know it'll be a bunch of `foo` type outputs. If its a `pd.DataFrame` we know its going to be `pd.Series`. Etc...
**Describe alternatives you've considered**
Not doing this.
**Additional context**
Just an idea that popped up when writing some new features.
Contributor guide
Research direction
Start by locating the existing `extract_fields` and `extract_columns` decorators and the code that registers or processes their outputs. Compare their current usage, then define the shared decorator contract and extension boundary described here. Done means dictionary and pandas DataFrame return forms work through the combined decorator without separately specifying the extracted type.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100