dask / dask/dask-expr

Support duplicated columns (maybe?)

Open
#778 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
89
Forks
26
PR merge metrics
No merged PRs in 30d

Description

We currently don't support duplicated columns, introducing support is not completely trivial since we would have to handle them properly in column projections. One example to explain the difficulties:

```
pdf = pd.DataFrame([[1, 2], [3, 4]], columns=list("aa"))
print(pdf["a"])

print(pdf[["a", "a"]])
```

```
0 1 2
1 3 4
a a a a
0 1 2 1 2
1 3 4 3 4
```

Selecting duplicated columns once returns all of them, but having them n times in the indexer duplicates the columns also n times, so we would have to be very careful to keep the count consistent. Other things also depend on set comparisons, which makes this even harder.

I don#t think that this is a very important feature for now, so I don't intend to prioritise this anytime soon

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the duplicated-column behavior in the pandas example from the issue, including both single-name and repeated-name selection. Trace how column projections and set comparisons handle the duplicated labels, then define tests that verify column counts remain consistent for each selection form. Done means duplicated columns are supported without changing the demonstrated selection semantics.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.