dask / dask/dask-expr

Method to check that two Dask Series have the same index

Open
#1,112 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
89
Forks
26
PR merge metrics
No merged PRs in 30d

Description

Hi - I discussed this a bit with @phofl , as I'm aiming to have zero-cost-abstraction around Dask DataFrame in [Narwhals](https://github.com/narwhals-dev/narwhals)

One thing I'd like to check is how to check that two Dask Series have the same index. Or, rather, that concatenating them would not result in any index alignment

Patrick pointed me to `dask_expr._expr.are_co_aligned`, which seemed to work great for me until I tried using `__getitem__`. Here's an example:
```python
In [21]: df = dd.from_pandas(pd.DataFrame({'a': [1,2,3], 'b': [4,5,6]}))

In [22]: dask_expr._expr.are_co_aligned(df._expr, df['a'][[1,2,0]]._expr)
Out[22]: True
```

This isn't quite what I was expecting - if I compute `df['a'][[1,2,0]]`, then the index has indeed been shuffled with respect to `df`

Is this a bug in `are_co_aligned`? If not, is there another way to check that index alignment does _not_ happen?

Thanks 🙏

Contributor guide

Open the contributing guide

Research direction

Reproduce the example using dask_expr._expr.are_co_aligned, df['a'][[1,2,0]], and a computed result. Read the implementation and related alignment tests to determine whether the reported result is expected, then define and verify a supported way to detect when concatenation will avoid index alignment.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.