apache / apache/arrow

Left join doesn't preserve order of left for right missing values

Open
#37,542 1 comment 0 reactions 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

I'm trying to do a `left outer` join (the default).

When all the keys in left can be found in right, the order of the left table is preserved:
```python
import pyarrow as pa

left = pa.table(
{
"key": ["KEY_1", "KEY_3"],
"left_value": ["1", "3"],
}
)

right = pa.table(
{
"key": ["KEY_1", "KEY_3"],
"right_value": ["1", "3"],
}
)
```

But if any key from left is missing in right, the order is changed. The missing values are put at the end:

```python
both = left.join(right, keys=["key"])
assert both["key"] == left["key"]

left = pa.table(
{
"key": ["KEY_1", "KEY_2", "KEY_3"],
"left_value": ["1", "2", "3"],
}
)

right = pa.table(
{
"key": ["KEY_1", "KEY_3"],
"right_value": ["1", "3"],
}
)

both = left.join(right, keys=["key"])
assert both["key"].to_pylist() == ["KEY_1", "KEY_3", "KEY_2"], "Should be 1,2,3"
```

I know there's no explicit guaranty on the order of the output data on the join, but it's weird that it preserve order when values are present, and changes it when they are missing.

Tested with `pyarrow==13.0.0`

### Component(s)

Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.