``col`` expression to replace callables as far as possible
- Dominant language
- Python
- Stars
- 89
- Forks
- 26
- PR merge metrics
- No merged PRs in 30d
Description
PySpark (https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.col.html) and Polars (https://pola-rs.github.io/polars/py-polars/html/reference/expressions/col.html) both have column expressions that enable users to use expressions instead of callable (FYI: we intend to add something similar in pandas). Short example:
```
df = from_pandas(pd.DataFrame({"a": [1, 2, 3], "b": [4, 5, 6], "c": 1, "d": 1, "e": 1}))
df.groupby("a"). transform(lambda: x: x.b / x.c.sum())
```
This is terrible from an optimization perspective, since the lambda doesn't tell us anything about what's going on in there. We could drop columns "d" and "e", but we have no way of knowing this.
If we add a col expression like Expression, we can rewrite this as follows:
```
import dask_expr as dx
df = from_pandas(pd.DataFrame({"a": [1, 2, 3], "b": [4, 5, 6], "c": 1, "d": 1, "e": 1}))
df.groupby("a").transform(dx.col("b") / dx.col("c").sum())
```
We could then look at the expression that replaces the callable and figure out how we can optimise our expression.
Not totally sure how the API should look like, since we have to inject the actual ``self.frame`` into the column expression ``dx.col("b") / dx.col("c").sum()`` inside of the transform step.
cc @mrocklin @rjzamora
Contributor guide
Research direction
The issue names no repository files or tests. Start by reviewing the existing Expression API and the PySpark and Polars col-expression references, then determine how transform should inject its frame. Done means a settled column-expression API can replace the callable example and expose enough structure for column pruning and other optimization.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100