[Bug] When using publish_time as a query condition in Pulsar SQL, the query result would lose one data record.
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 160
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### Version
master
### Minimal reproduce step
Raw data:
```
68 65 6c 6c 6f 30 | -1 | NULL | 2023-08-02 07:33:15.274 | (449,0,0) | 0 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 31 | -1 | NULL | 2023-08-02 07:33:25.305 | (449,1,0) | 1 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 32 | -1 | NULL | 2023-08-02 07:33:35.314 | (449,2,0) | 2 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 33 | -1 | NULL | 2023-08-02 07:33:45.324 | (449,3,0) | 3 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 34 | -1 | NULL | 2023-08-02 07:33:55.336 | (449,4,0) | 4 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 35 | -1 | NULL | 2023-08-02 07:34:05.345 | (449,5,0) | 5 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 36 | -1 | NULL | 2023-08-02 07:34:15.355 | (449,6,0) | 6 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 37 | -1 | NULL | 2023-08-02 07:34:25.395 | (449,7,0) | 7 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 38 | -1 | NULL | 2023-08-02 07:34:35.404 | (449,8,0) | 8 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 39 | -1 | NULL | 2023-08-02 07:34:45.412 | (449,9,0) | 9 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 30 | -1 | NULL | 2023-08-02 07:46:50.961 | (449,10,0) | 0 | standalone-8-2 | NULL | {}
```
SQL:
```sql
select * from pulsar."public/default"."topic10" WHERE "__publish_time__" > TIMESTAMP '2023-08-02 07:33:35' AND "__publish_time__" < TIMESTAMP '2023-08-02 07:46:51';
```
Result:
```
68 65 6c 6c 6f 32 | -1 | NULL | 2023-08-02 07:33:35.314 | (449,2,0) | 2 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 33 | -1 | NULL | 2023-08-02 07:33:45.324 | (449,3,0) | 3 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 34 | -1 | NULL | 2023-08-02 07:33:55.336 | (449,4,0) | 4 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 35 | -1 | NULL | 2023-08-02 07:34:05.345 | (449,5,0) | 5 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 36 | -1 | NULL | 2023-08-02 07:34:15.355 | (449,6,0) | 6 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 37 | -1 | NULL | 2023-08-02 07:34:25.395 | (449,7,0) | 7 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 38 | -1 | NULL | 2023-08-02 07:34:35.404 | (449,8,0) | 8 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 39 | -1 | NULL | 2023-08-02 07:34:45.412 | (449,9,0) | 9 | standalone-8-1 | NULL | {}
(8 rows)
```
### What did you expect to see?
```
68 65 6c 6c 6f 32 | -1 | NULL | 2023-08-02 07:33:35.314 | (449,2,0) | 2 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 33 | -1 | NULL | 2023-08-02 07:33:45.324 | (449,3,0) | 3 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 34 | -1 | NULL | 2023-08-02 07:33:55.336 | (449,4,0) | 4 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 35 | -1 | NULL | 2023-08-02 07:34:05.345 | (449,5,0) | 5 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 36 | -1 | NULL | 2023-08-02 07:34:15.355 | (449,6,0) | 6 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 37 | -1 | NULL | 2023-08-02 07:34:25.395 | (449,7,0) | 7 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 38 | -1 | NULL | 2023-08-02 07:34:35.404 | (449,8,0) | 8 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 39 | -1 | NULL | 2023-08-02 07:34:45.412 | (449,9,0) | 9 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 30 | -1 | NULL | 2023-08-02 07:46:50.961 | (449,10,0) | 0 | standalone-8-2 | NULL | {}
(9 rows)
```
### What did you see instead?
The missing data record:
```
68 65 6c 6c 6f 30 | -1 | NULL | 2023-08-02 07:46:50.961 | (449,10,0) | 0 | standalone-8-2 | NULL | {}
```
### Anything else?
https://github.com/apache/pulsar/blob/782e91fe327efe2c9c9107d6c679c2837d43935b/pulsar-sql/presto-pulsar/src/main/java/org/apache/pulsar/sql/presto/PulsarSplitManager.java#L420-L423
I discovered that the query range reduced 1 at the last, after removing it, the running looks well.
Why `-1`, any special considerations?
### Are you willing to submit a PR?
- [X] I'm willing to submit a PR!
Contributor guide
Research direction
Start at pulsar-sql/presto-pulsar/src/main/java/org/apache/pulsar/sql/presto/PulsarSplitManager.java around lines 420-423 and reproduce the reported Pulsar SQL query with the supplied publish times. Trace why the upper query range is reduced by one and verify the fix by confirming that all 9 expected records, including the final record, are returned.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, sql
- Domain
- backend, stream-processing
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100