apache / apache/datafusion

offset command not working as expected

Open
#9,488 3 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

Giving wrong results for large rows

### To Reproduce

If I have data of less than 753459 rows it will give this result. this is data for `753458` rows
┌────────┬────────┐
│ data │ index │
│ int64 │ int64 │
├────────┼────────┤
│ 1 │ 0 │
│ 2 │ 1 │
│ 3 │ 2 │
│ 4 │ 3 │
│ 5 │ 4 │
│ 6 │ 5 │
│ 7 │ 6 │
│ 8 │ 7 │
│ 9 │ 8 │
│ 10 │ 9 │
│ 11 │ 10 │
│ 12 │ 11 │
│ 13 │ 12 │
│ 14 │ 13 │
│ 15 │ 14 │
│ 16 │ 15 │
│ 17 │ 16 │
│ 18 │ 17 │
│ 19 │ 18 │
│ 20 │ 19 │
│ · │ · │
│ · │ · │
│ · │ · │
│ 793439 │ 793438 │
│ 793440 │ 793439 │
│ 793441 │ 793440 │
│ 793442 │ 793441 │
│ 793443 │ 793442 │
│ 793444 │ 793443 │
│ 793445 │ 793444 │
│ 793446 │ 793445 │
│ 793447 │ 793446 │
│ 793448 │ 793447 │
│ 793449 │ 793448 │
│ 793450 │ 793449 │
│ 793451 │ 793450 │
│ 793452 │ 793451 │
│ 793453 │ 793452 │
│ 793454 │ 793453 │
│ 793455 │ 793454 │
│ 793456 │ 793455 │
│ 793457 │ 793456 │
│ 793458 │ 793457 │
├────────┴────────┤
│ 793458 rows │
│ (40 shown)
```

datafusion-cli -q -c " select * from 'offset.csv' limit 4 offset 3"
data,index
4,3
5,4
6,5
7,6
```
If the data is more it gives another
this is data for `793459` rows
```
datafusion-cli -q -c " select * from 'offset.csv' limit 4 offset 3"
data,index
113075,113074
113076,113075
113077,113076
113078,113077
```
for `793460` rows
```
datafusion-cli -q -c " select * from 'offset.csv' limit 4 offset 3"
data,index
696266,696265
696267,696266
696268,696267
696269,696268
```

### Expected behavior

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Reproduce the issue with the datafusion-cli query against offset.csv, varying the CSV size around 793,458 to 793,460 rows. Trace the LIMIT/OFFSET execution path and compare the returned rows with the expected indexes 3 through 6. Done means the same four rows are returned consistently for each input size.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.