mars-project / mars-project/mars

[BUG] Behavior of creating DataFrame from dicts not consistent with pandas

Open
#1,727 0 comments 0 reactions 0 assignees View on GitHub
mod: dataframe type: bug
Dominant language
Python
Stars
2.7k
Forks
325
PR merge metrics
No merged PRs in 30d

Description

**Describe the bug**
When creating Mars DataFrame from dicts with Series and supply an extra index, pandas will reindex the series using supplied index object, while Mars merely replaces the whole index.

All DataFrame / Series initializers with extra indices need to be inspected.

**To Reproduce**
```python
import numpy as np
import pandas as pd

import mars.dataframe as md
import mars.tensor as mt

raw_dict = {
'C': np.random.choice(['u', 'v', 'w'], size=(100,)),
'A': pd.Series(np.random.rand(100)),
'B': np.random.randint(0, 10, size=(100,)),
}

idx_array = np.arange(0, 100)
np.random.shuffle(idx_array)
raw_idx = pd.Index(idx_array, name='test_index')
r = md.DataFrame(raw_dict, columns=list('ABC'), index=md.Index(raw_idx)).execute()
# assertion error here
pd.testing.assert_frame_equal(r, pd.DataFrame(raw_dict, columns=list('ABC'), index=raw_idx))
```

Contributor guide

Open the contributing guide

Research direction

Start by running the provided reproduction and comparing Mars DataFrame construction with pandas when a Series and extra index are supplied. Inspect the DataFrame and Series initializers for similar extra-index handling; done when the assertion passes and related initializers match pandas behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, pandas, python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.