apache / apache/pulsar

[Bug] When using publish_time as a query condition in Pulsar SQL, the query result would lose one data record.

Open
#20,919 1 comment 0 reactions 0 assignees View on GitHub
Stale type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### Version

master

### Minimal reproduce step

Raw data:
```
68 65 6c 6c 6f 30 | -1 | NULL | 2023-08-02 07:33:15.274 | (449,0,0) | 0 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 31 | -1 | NULL | 2023-08-02 07:33:25.305 | (449,1,0) | 1 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 32 | -1 | NULL | 2023-08-02 07:33:35.314 | (449,2,0) | 2 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 33 | -1 | NULL | 2023-08-02 07:33:45.324 | (449,3,0) | 3 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 34 | -1 | NULL | 2023-08-02 07:33:55.336 | (449,4,0) | 4 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 35 | -1 | NULL | 2023-08-02 07:34:05.345 | (449,5,0) | 5 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 36 | -1 | NULL | 2023-08-02 07:34:15.355 | (449,6,0) | 6 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 37 | -1 | NULL | 2023-08-02 07:34:25.395 | (449,7,0) | 7 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 38 | -1 | NULL | 2023-08-02 07:34:35.404 | (449,8,0) | 8 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 39 | -1 | NULL | 2023-08-02 07:34:45.412 | (449,9,0) | 9 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 30 | -1 | NULL | 2023-08-02 07:46:50.961 | (449,10,0) | 0 | standalone-8-2 | NULL | {}
```

SQL:
```sql
select * from pulsar."public/default"."topic10" WHERE "__publish_time__" > TIMESTAMP '2023-08-02 07:33:35' AND "__publish_time__" < TIMESTAMP '2023-08-02 07:46:51';
```

Result:
```
68 65 6c 6c 6f 32 | -1 | NULL | 2023-08-02 07:33:35.314 | (449,2,0) | 2 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 33 | -1 | NULL | 2023-08-02 07:33:45.324 | (449,3,0) | 3 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 34 | -1 | NULL | 2023-08-02 07:33:55.336 | (449,4,0) | 4 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 35 | -1 | NULL | 2023-08-02 07:34:05.345 | (449,5,0) | 5 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 36 | -1 | NULL | 2023-08-02 07:34:15.355 | (449,6,0) | 6 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 37 | -1 | NULL | 2023-08-02 07:34:25.395 | (449,7,0) | 7 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 38 | -1 | NULL | 2023-08-02 07:34:35.404 | (449,8,0) | 8 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 39 | -1 | NULL | 2023-08-02 07:34:45.412 | (449,9,0) | 9 | standalone-8-1 | NULL | {}
(8 rows)
```

### What did you expect to see?

```
68 65 6c 6c 6f 32 | -1 | NULL | 2023-08-02 07:33:35.314 | (449,2,0) | 2 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 33 | -1 | NULL | 2023-08-02 07:33:45.324 | (449,3,0) | 3 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 34 | -1 | NULL | 2023-08-02 07:33:55.336 | (449,4,0) | 4 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 35 | -1 | NULL | 2023-08-02 07:34:05.345 | (449,5,0) | 5 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 36 | -1 | NULL | 2023-08-02 07:34:15.355 | (449,6,0) | 6 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 37 | -1 | NULL | 2023-08-02 07:34:25.395 | (449,7,0) | 7 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 38 | -1 | NULL | 2023-08-02 07:34:35.404 | (449,8,0) | 8 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 39 | -1 | NULL | 2023-08-02 07:34:45.412 | (449,9,0) | 9 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 30 | -1 | NULL | 2023-08-02 07:46:50.961 | (449,10,0) | 0 | standalone-8-2 | NULL | {}
(9 rows)
```

### What did you see instead?

The missing data record:
```
68 65 6c 6c 6f 30 | -1 | NULL | 2023-08-02 07:46:50.961 | (449,10,0) | 0 | standalone-8-2 | NULL | {}
```

### Anything else?

https://github.com/apache/pulsar/blob/782e91fe327efe2c9c9107d6c679c2837d43935b/pulsar-sql/presto-pulsar/src/main/java/org/apache/pulsar/sql/presto/PulsarSplitManager.java#L420-L423

I discovered that the query range reduced 1 at the last, after removing it, the running looks well.
Why `-1`, any special considerations?

### Are you willing to submit a PR?

- [X] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start at pulsar-sql/presto-pulsar/src/main/java/org/apache/pulsar/sql/presto/PulsarSplitManager.java around lines 420-423 and reproduce the reported Pulsar SQL query with the supplied publish times. Trace why the upper query range is reduced by one and verify the fix by confirming that all 9 expected records, including the final record, are returned.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
backend, stream-processing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.