offset command not working as expected
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Describe the bug
Giving wrong results for large rows
### To Reproduce
If I have data of less than 753459 rows it will give this result. this is data for `753458` rows
┌────────┬────────┐
│ data │ index │
│ int64 │ int64 │
├────────┼────────┤
│ 1 │ 0 │
│ 2 │ 1 │
│ 3 │ 2 │
│ 4 │ 3 │
│ 5 │ 4 │
│ 6 │ 5 │
│ 7 │ 6 │
│ 8 │ 7 │
│ 9 │ 8 │
│ 10 │ 9 │
│ 11 │ 10 │
│ 12 │ 11 │
│ 13 │ 12 │
│ 14 │ 13 │
│ 15 │ 14 │
│ 16 │ 15 │
│ 17 │ 16 │
│ 18 │ 17 │
│ 19 │ 18 │
│ 20 │ 19 │
│ · │ · │
│ · │ · │
│ · │ · │
│ 793439 │ 793438 │
│ 793440 │ 793439 │
│ 793441 │ 793440 │
│ 793442 │ 793441 │
│ 793443 │ 793442 │
│ 793444 │ 793443 │
│ 793445 │ 793444 │
│ 793446 │ 793445 │
│ 793447 │ 793446 │
│ 793448 │ 793447 │
│ 793449 │ 793448 │
│ 793450 │ 793449 │
│ 793451 │ 793450 │
│ 793452 │ 793451 │
│ 793453 │ 793452 │
│ 793454 │ 793453 │
│ 793455 │ 793454 │
│ 793456 │ 793455 │
│ 793457 │ 793456 │
│ 793458 │ 793457 │
├────────┴────────┤
│ 793458 rows │
│ (40 shown)
```
datafusion-cli -q -c " select * from 'offset.csv' limit 4 offset 3"
data,index
4,3
5,4
6,5
7,6
```
If the data is more it gives another
this is data for `793459` rows
```
datafusion-cli -q -c " select * from 'offset.csv' limit 4 offset 3"
data,index
113075,113074
113076,113075
113077,113076
113078,113077
```
for `793460` rows
```
datafusion-cli -q -c " select * from 'offset.csv' limit 4 offset 3"
data,index
696266,696265
696267,696266
696268,696267
696269,696268
```
### Expected behavior
_No response_
### Additional context
_No response_
Contributor guide
Research direction
Reproduce the issue with the datafusion-cli query against offset.csv, varying the CSV size around 793,458 to 793,460 rows. Trace the LIMIT/OFFSET execution path and compare the returned rows with the expected indexes 3 through 6. Done means the same four rows are returned consistently for each input size.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100