Some thoughts on high-level reasoning
- Dominant language
- Python
- Stars
- 89
- Forks
- 26
- PR merge metrics
- No merged PRs in 30d
Description
I had two conversations recently with @fjetter and @phofl about reasoning about dataframes. Often during different stages (expression building, optimization, task graph building) we want to know various attributes of our dataframe. Here are a few things people have asked for in the past:
1. Is it sorted along this column?
2. How many partitions do we have?
3. How many rows are in each partition?
4. What are the min/max values of each partition in a sorted dataframe so that we can do joins better?
Historically we relied on `divisions` to answer many of these questions, and we're still used to doing that. However, this isn't great, because divisions can change as we optimize (maybe we decide to repartition based on worker size, for example) and because divisions doesn't hold all of the infomation we want (like lengths of partitions).
Instead, I think that we need to get used to asking questions of expressions by either inspecting the current tree, or by using other expressions. Two examples:
1. @rjzamora added a `Lengths` expression, which computes the length of each partition, and knows how to optimize itself through Elemwise and ReadParquet calls so that, oftentimes, we can convert a `Lengths(df)` expression into a `Literal`.
This is an example of using an expression to answer a question about an existing expression
2. We could ask if two dataframes share the same partition alignment by asking if they have the same ancestors modulo `Blockwise`. For example the following are co-aligned:
```python
df = read_parquet(...)
a = df.x[df.z > 1] + 10
b = df.y - df.z.sum()
assert are_co_aligned(a, b)
```
This is an example of looking at an expression tree to answer a question.
3. Similar to above we might infer that a column is sorted if it had previously been the target of a `set_index` or `sort_values` call, and was only operated on by monotonic expressions (we'd have to add this to known monotonic expressions)
General principles:
1. rely on divsiions less
2. try to only capture user inputs in Expression operands
3. think about asking questions using expression structure, or with new expressions with fun optimizations
Contributor guide
Assessment
This issue has not been assessed yet.