dask / dask/dask-expr

``col`` expression to replace callables as far as possible

Open
#386 4 comments 2 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
89
Forks
26
PR merge metrics
No merged PRs in 30d

Description

PySpark (https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.col.html) and Polars (https://pola-rs.github.io/polars/py-polars/html/reference/expressions/col.html) both have column expressions that enable users to use expressions instead of callable (FYI: we intend to add something similar in pandas). Short example:

```
df = from_pandas(pd.DataFrame({"a": [1, 2, 3], "b": [4, 5, 6], "c": 1, "d": 1, "e": 1}))
df.groupby("a"). transform(lambda: x: x.b / x.c.sum())
```

This is terrible from an optimization perspective, since the lambda doesn't tell us anything about what's going on in there. We could drop columns "d" and "e", but we have no way of knowing this.

If we add a col expression like Expression, we can rewrite this as follows:

```
import dask_expr as dx

df = from_pandas(pd.DataFrame({"a": [1, 2, 3], "b": [4, 5, 6], "c": 1, "d": 1, "e": 1}))
df.groupby("a").transform(dx.col("b") / dx.col("c").sum())
```

We could then look at the expression that replaces the callable and figure out how we can optimise our expression.

Not totally sure how the API should look like, since we have to inject the actual ``self.frame`` into the column expression ``dx.col("b") / dx.col("c").sum()`` inside of the transform step.

cc @mrocklin @rjzamora

Contributor guide

Open the contributing guide

Research direction

The issue names no repository files or tests. Start by reviewing the existing Expression API and the PySpark and Polars col-expression references, then determine how transform should inject its frame. Done means a settled column-expression API can replace the callable example and expose enough structure for column pruning and other optimization.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.