apache / apache/pinot

H3 index bug

Open
#7,766 4 comments 0 reactions 0 assignees View on GitHub
bug geo
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

When using the H3 index, in some cases, the correct filtering of distances is not possible.

## Test case
### Data

#### geodata.csv
```csv
name,lng,lat
a,-103.34056886616187,20.63611463218247
b,-103.31104310932645,20.606231326034603
c,-103.33524736347661,20.595626162929634
```
#### schema
```json
{
"metricFieldSpecs": [],
"dimensionFieldSpecs": [
{
"dataType": "STRING",
"name": "name"
},
{
"dataType": "FLOAT",
"name": "lng"
},
{
"dataType": "FLOAT",
"name": "lat"
},
{
"dataType": "BYTES",
"name": "location_st_point",
"transformFunction": "toSphericalGeography(stPoint(lng,lat))"
}
],
"schemaName": "geodata"
}
```

#### table

```json
{
"tableName": "geodata",
"tableType": "OFFLINE",
"segmentsConfig": {
"retentionTimeUnit": "DAYS",
"retentionTimeValue": "1",
"segmentPushType": "APPEND",
"segmentAssignmentStrategy": "BalanceNumSegmentAssignmentStrategy",
"schemaName": "geodata",
"replication": "1"
},
"tenants": {
},
"fieldConfigList": [
{
"name": "location_st_point",
"encodingType": "RAW",
"indexType": "H3",
"properties": {
"resolutions": "7"
}
}
],
"tableIndexConfig": {
"loadMode": "MMAP",
"noDictionaryColumns": [
"location_st_point"
]
},
"metadata": {
"customConfigs": {
}
}
}
```

### Query
```sql
select name,
lat,
lng,
ST_DISTANCE(
location_st_point,
ST_Point(-103.34417375507813, 20.64061268636347, 1)
) as distance
from geodata
where ST_DISTANCE(
location_st_point,
ST_Point(-103.34417375507813, 20.64061268636347, 1)
) < 5000;
```
### Query Response
```csv
name,lat,lng,distance
a,20.636114,-103.34057,625.2019871997669
b,20.606232,-103.31104,5148.1557916677875
c,20.595627,-103.33525,5087.813601469894
```

## Analysis
**This appears to be a problem with KRing calculations***

### Code
https://github.com/apache/pinot/blob/e3d238ac1d8633331d9507713266e41e6b40f870/pinot-core/src/main/java/org/apache/pinot/core/operator/filter/H3IndexFilterOperator.java#L186-L198

### Visualize
![poc](https://user-images.githubusercontent.com/13296462/141759466-bc0639cc-0c79-4ea3-8c2e-f28f3d5aa552.png)

Contributor guide

Open the contributing guide

Research direction

Start in pinot-core/src/main/java/org/apache/pinot/core/operator/filter/H3IndexFilterOperator.java at the linked KRing calculation around lines 186-198. Reproduce the supplied geodata schema, H3 resolution 7, and ST_DISTANCE query, then inspect how the candidate cells are selected. Done means the query returns only rows within 5000 meters while preserving the expected nearby result.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.