apache / apache/datafusion

Document non-deterministic ordering of joining, group by, etc.

Open
#7,373 8 comments 0 reactions 0 assignees View on GitHub
documentation
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

I am using the SQL interface to query parquet data. I am registering each file in a datafusion context. The query contains some joins, and groupbys (where I speculate the trouble is). Maybe 1/5 attempts I get the correct answer from a compiled binary (0 records). but the other 4/5 attempts I see a lot of erroneous results appearing.

So to be clear, the correct answer is 0 records, and we should never see records appearing otherwise (unless DF's groupby operations are nondeterministic/nonsequential?). Yet I only see that on a rare occasion of runs.

I feel like I might be missing something here(do I need to sort first?) but this looks like a bug to me.

### To Reproduce

I can't share the data, but the query looks like the last two queries on this sql fiddle(they're the same) I borrowed from someone on stack overflow and wrote the query I care about.

https://dbfiddle.uk/hA8-ejaw

In this example we do see some rows being returned but in my actual use case there should be none.

### Expected behavior

The correct answer of records are returned or there is documentation explaining why this doesn't happen/an example?

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with the SQL interface and the final two queries in the linked dbfiddle example, then investigate whether joins and GROUP BY guarantee row ordering or deterministic results. Done means documenting the relevant ordering behavior and providing a reproducible example or explanation of why the reported result cannot be guaranteed.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
databases
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.