apache / apache/pinot

NullPointerException when creating a Segment where star-tree index refers to invalid column

Open
#5,899 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

Just a usability thing: when writing out some configuration, I had a typo in a schema field name so the star tree configuration did not have a list of all valid columns. It took a bit of poking to eventually realize the mistake. For example:

schema
```json
{
"schemaName": "testdata",
"dimensionFieldSpecs": [{
"name": "transaction_id",
"dataType": "STRING"
}],
"metricFieldSpecs": [{
"name": "amount",
"dataType": "DOUBLE"
}],
"dateTimeFieldSpecs": [{
"name": "created_at",
"dataType": "LONG",
"format": "1:SECONDS:EPOCH",
"granularity": "15:MINUTES"
}]
}
```

table
```json
{
"tableName": "testdata",
"tableType": "OFFLINE",
"routing": {
"segmentPrunerType": "partition"
},
"segmentsConfig": {
"timeColumnName": "created_at",
"timeType": "SECONDS",
"replication": "1",
"schemaName": "testdata",
"segmentPushFrequency": "HOURLY",
"segmentPushType": "APPEND"
},
"tableIndexConfig": {
"loadMode": "MMAP",
"starTreeIndexConfigs": [{
"dimensionsSplitOrder": [
"does_not_exist"
],
"functionColumnPairs": [
"COUNT"
],
"maxLeafRecords": 1
}]
},
"tenants": {},
"metadata": {}
}
```

Using `pinot-admin.sh CreateSegment`, this is the exception
```
Exception caught:
java.util.concurrent.ExecutionException: java.lang.NullPointerException
at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:?]
at java.util.concurrent.FutureTask.get(FutureTask.java:191) ~[?:?]
at org.apache.pinot.tools.admin.command.CreateSegmentCommand.execute(CreateSegmentCommand.java:264) ~[pinot-all-0.5.0-2020-08-13-SNAPSHOT-jar-with-dependencies.jar:0.5.0-2020-08-13-SNAPSHOT-5ac67eb1a1210cb6c0c76d81d138cac29d0d3d4e]
at org.apache.pinot.tools.admin.PinotAdministrator.execute(PinotAdministrator.java:150) [pinot-all-0.5.0-2020-08-13-SNAPSHOT-jar-with-dependencies.jar:0.5.0-2020-08-13-SNAPSHOT-5ac67eb1a1210cb6c0c76d81d138cac29d0d3d4e]
at org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:162) [pinot-all-0.5.0-2020-08-13-SNAPSHOT-jar-with-dependencies.jar:0.5.0-2020-08-13-SNAPSHOT-5ac67eb1a1210cb6c0c76d81d138cac29d0d3d4e]
Caused by: java.lang.NullPointerException
at org.apache.pinot.core.segment.index.datasource.ImmutableDataSource$ImmutableDataSourceMetadata.(ImmutableDataSource.java:54) ~[pinot-all-0.5.0-2020-08-13-SNAPSHOT-jar-with-dependencies.jar:0.5.0-2020-08-13-SNAPSHOT-5ac67eb1a1210cb6c0c76d81d138cac29d0d3d4e]
at org.apache.pinot.core.segment.index.datasource.ImmutableDataSource.(ImmutableDataSource.java:36) ~[pinot-all-0.5.0-2020-08-13-SNAPSHOT-jar-with-dependencies.jar:0.5.0-2020-08-13-SNAPSHOT-5ac67eb1a1210cb6c0c76d81d138cac29d0d3d4e]
at org.apache.pinot.core.indexsegment.immutable.ImmutableSegmentImpl.getDataSource(ImmutableSegmentImpl.java:94) ~[pinot-all-0.5.0-2020-08-13-SNAPSHOT-jar-with-dependencies.jar:0.5.0-2020-08-13-SNAPSHOT-5ac67eb1a1210cb6c0c76d81d138cac29d0d3d4e]
at org.apache.pinot.core.data.readers.PinotSegmentColumnReader.(PinotSegmentColumnReader.java:38) ~[pinot-all-0.5.0-2020-08-13-SNAPSHOT-jar-with-dependencies.jar:0.5.0-2020-08-13-SNAPSHOT-5ac67eb1a1210cb6c0c76d81d138cac29d0d3d4e]
at org.apache.pinot.core.startree.v2.builder.BaseSingleTreeBuilder.(BaseSingleTreeBuilder.java:129) ~[pinot-all-0.5.0-2020-08-13-SNAPSHOT-jar-with-dependencies.jar:0.5.0-2020-08-13-SNAPSHOT-5ac67eb1a1210cb6c0c76d81d138cac29d0d3d4e]
at org.apache.pinot.core.startree.v2.builder.OffHeapSingleTreeBuilder.(OffHeapSingleTreeBuilder.java:69) ~[pinot-all-0.5.0-2020-08-13-SNAPSHOT-jar-with-dependencies.jar:0.5.0-2020-08-13-SNAPSHOT-5ac67eb1a1210cb6c0c76d81d138cac29d0d3d4e]
at org.apache.pinot.core.startree.v2.builder.MultipleTreesBuilder.getSingleTreeBuilder(MultipleTreesBuilder.java:157) ~[pinot-all-0.5.0-2020-08-13-SNAPSHOT-jar-with-dependencies.jar:0.5.0-2020-08-13-SNAPSHOT-5ac67eb1a1210cb6c0c76d81d138cac29d0d3d4e]
at org.apache.pinot.core.startree.v2.builder.MultipleTreesBuilder.build(MultipleTreesBuilder.java:130) ~[pinot-all-0.5.0-2020-08-13-SNAPSHOT-jar-with-dependencies.jar:0.5.0-2020-08-13-SNAPSHOT-5ac67eb1a1210cb6c0c76d81d138cac29d0d3d4e]
at org.apache.pinot.core.segment.creator.impl.SegmentIndexCreationDriverImpl.buildStarTreeV2IfNecessary(SegmentIndexCreationDriverImpl.java:298) ~[pinot-all-0.5.0-2020-08-13-SNAPSHOT-jar-with-dependencies.jar:0.5.0-2020-08-13-SNAPSHOT-5ac67eb1a1210cb6c0c76d81d138cac29d0d3d4e]
at org.apache.pinot.core.segment.creator.impl.SegmentIndexCreationDriverImpl.handlePostCreation(SegmentIndexCreationDriverImpl.java:263) ~[pinot-all-0.5.0-2020-08-13-SNAPSHOT-jar-with-dependencies.jar:0.5.0-2020-08-13-SNAPSHOT-5ac67eb1a1210cb6c0c76d81d138cac29d0d3d4e]
at org.apache.pinot.core.segment.creator.impl.SegmentIndexCreationDriverImpl.build(SegmentIndexCreationDriverImpl.java:223) ~[pinot-all-0.5.0-2020-08-13-SNAPSHOT-jar-with-dependencies.jar:0.5.0-2020-08-13-SNAPSHOT-5ac67eb1a1210cb6c0c76d81d138cac29d0d3d4e]
at org.apache.pinot.tools.admin.command.CreateSegmentCommand.lambda$execute$0(CreateSegmentCommand.java:239) ~[pinot-all-0.5.0-2020-08-13-SNAPSHOT-jar-with-dependencies.jar:0.5.0-2020-08-13-SNAPSHOT-5ac67eb1a1210cb6c0c76d81d138cac29d0d3d4e]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) ~[?:?]
at java.lang.Thread.run(Thread.java:832) ~[?:?]
```

Contributor guide

Open the contributing guide

Research direction

Reproduce the failure with the schema and table configurations using pinot-admin.sh CreateSegment. Start with CreateSegmentCommand.java and the stack-trace path through ImmutableDataSource.java and BaseSingleTreeBuilder.java; done means an invalid star-tree column no longer produces an unexplained NullPointerException.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.