dask / dask/dask

Flaky `test_collection.py::test_len`

Open
#12,275 9 comments 0 reactions 0 assignees View on GitHub
tests
Dominant language
Python
Stars
13.9k
Forks
2k
PR merge metrics
No merged PRs in 30d

Description

This test fails on both pandas 2.3 and 3.0 CI with some frequency:

https://github.com/dask/dask/actions/runs/21682277456/job/62519785526?pr=12271

```
def test_len(df, pdf):
df2 = df[["x"]] + 1
assert len(df2) == len(pdf)

assert len(df[df.x > 5]) == len(pdf[pdf.x > 5])

first = df2.partitions[0].compute()
> assert len(df2.partitions[0]) == len(first)
E AssertionError: assert 100 == 10
E + where 100 = len(Dask DataFrame Structure:\n x\nnpartitions=1 \n0 int64\n10 ...\nDask Name: partitions, 4 expressions\nExpr=Partitions(frame=df[['x']] + 1, partitions=[0]))
E + and 10 = len( x\n0 1\n1 2\n2 3\n3 4\n4 5\n5 6\n6 7\n7 8\n8 9\n9 10)

dask/dataframe/dask_expr/tests/test_collection.py:1718: AssertionError

FAILED dask/dataframe/dask_expr/tests/test_collection.py::test_len - AssertionError: assert 100 == 10
+ where 100 = len(Dask DataFrame Structure:\n x\nnpartitions=1 \n0 int64\n10 ...\nDask Name: partitions, 4 expressions\nExpr=Partitions(frame=df[['x']] + 1, partitions=[0]))
+ and 10 = len( x\n0 1\n1 2\n2 3\n3 4\n4 5\n5 6\n6 7\n7 8\n8 9\n9 10)
```

Contributor guide

Open the contributing guide

Research direction

Start with dask/dataframe/dask_expr/tests/test_collection.py::test_len and reproduce the failure on the pandas 2.3 and 3.0 CI environments referenced in the issue. Compare the lazy partition length with the computed partition length around the failing assertion. Done means the test is stable and the partition length assertion consistently matches the computed result.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data, testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.