NullPointerException when creating a Segment where star-tree index refers to invalid column
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 189
Description
Just a usability thing: when writing out some configuration, I had a typo in a schema field name so the star tree configuration did not have a list of all valid columns. It took a bit of poking to eventually realize the mistake. For example:
schema
```json
{
"schemaName": "testdata",
"dimensionFieldSpecs": [{
"name": "transaction_id",
"dataType": "STRING"
}],
"metricFieldSpecs": [{
"name": "amount",
"dataType": "DOUBLE"
}],
"dateTimeFieldSpecs": [{
"name": "created_at",
"dataType": "LONG",
"format": "1:SECONDS:EPOCH",
"granularity": "15:MINUTES"
}]
}
```
table
```json
{
"tableName": "testdata",
"tableType": "OFFLINE",
"routing": {
"segmentPrunerType": "partition"
},
"segmentsConfig": {
"timeColumnName": "created_at",
"timeType": "SECONDS",
"replication": "1",
"schemaName": "testdata",
"segmentPushFrequency": "HOURLY",
"segmentPushType": "APPEND"
},
"tableIndexConfig": {
"loadMode": "MMAP",
"starTreeIndexConfigs": [{
"dimensionsSplitOrder": [
"does_not_exist"
],
"functionColumnPairs": [
"COUNT"
],
"maxLeafRecords": 1
}]
},
"tenants": {},
"metadata": {}
}
```
Using `pinot-admin.sh CreateSegment`, this is the exception
```
Exception caught:
java.util.concurrent.ExecutionException: java.lang.NullPointerException
at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:?]
at java.util.concurrent.FutureTask.get(FutureTask.java:191) ~[?:?]
at org.apache.pinot.tools.admin.command.CreateSegmentCommand.execute(CreateSegmentCommand.java:264) ~[pinot-all-0.5.0-2020-08-13-SNAPSHOT-jar-with-dependencies.jar:0.5.0-2020-08-13-SNAPSHOT-5ac67eb1a1210cb6c0c76d81d138cac29d0d3d4e]
at org.apache.pinot.tools.admin.PinotAdministrator.execute(PinotAdministrator.java:150) [pinot-all-0.5.0-2020-08-13-SNAPSHOT-jar-with-dependencies.jar:0.5.0-2020-08-13-SNAPSHOT-5ac67eb1a1210cb6c0c76d81d138cac29d0d3d4e]
at org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:162) [pinot-all-0.5.0-2020-08-13-SNAPSHOT-jar-with-dependencies.jar:0.5.0-2020-08-13-SNAPSHOT-5ac67eb1a1210cb6c0c76d81d138cac29d0d3d4e]
Caused by: java.lang.NullPointerException
at org.apache.pinot.core.segment.index.datasource.ImmutableDataSource$ImmutableDataSourceMetadata.(ImmutableDataSource.java:54) ~[pinot-all-0.5.0-2020-08-13-SNAPSHOT-jar-with-dependencies.jar:0.5.0-2020-08-13-SNAPSHOT-5ac67eb1a1210cb6c0c76d81d138cac29d0d3d4e]
at org.apache.pinot.core.segment.index.datasource.ImmutableDataSource.(ImmutableDataSource.java:36) ~[pinot-all-0.5.0-2020-08-13-SNAPSHOT-jar-with-dependencies.jar:0.5.0-2020-08-13-SNAPSHOT-5ac67eb1a1210cb6c0c76d81d138cac29d0d3d4e]
at org.apache.pinot.core.indexsegment.immutable.ImmutableSegmentImpl.getDataSource(ImmutableSegmentImpl.java:94) ~[pinot-all-0.5.0-2020-08-13-SNAPSHOT-jar-with-dependencies.jar:0.5.0-2020-08-13-SNAPSHOT-5ac67eb1a1210cb6c0c76d81d138cac29d0d3d4e]
at org.apache.pinot.core.data.readers.PinotSegmentColumnReader.(PinotSegmentColumnReader.java:38) ~[pinot-all-0.5.0-2020-08-13-SNAPSHOT-jar-with-dependencies.jar:0.5.0-2020-08-13-SNAPSHOT-5ac67eb1a1210cb6c0c76d81d138cac29d0d3d4e]
at org.apache.pinot.core.startree.v2.builder.BaseSingleTreeBuilder.(BaseSingleTreeBuilder.java:129) ~[pinot-all-0.5.0-2020-08-13-SNAPSHOT-jar-with-dependencies.jar:0.5.0-2020-08-13-SNAPSHOT-5ac67eb1a1210cb6c0c76d81d138cac29d0d3d4e]
at org.apache.pinot.core.startree.v2.builder.OffHeapSingleTreeBuilder.(OffHeapSingleTreeBuilder.java:69) ~[pinot-all-0.5.0-2020-08-13-SNAPSHOT-jar-with-dependencies.jar:0.5.0-2020-08-13-SNAPSHOT-5ac67eb1a1210cb6c0c76d81d138cac29d0d3d4e]
at org.apache.pinot.core.startree.v2.builder.MultipleTreesBuilder.getSingleTreeBuilder(MultipleTreesBuilder.java:157) ~[pinot-all-0.5.0-2020-08-13-SNAPSHOT-jar-with-dependencies.jar:0.5.0-2020-08-13-SNAPSHOT-5ac67eb1a1210cb6c0c76d81d138cac29d0d3d4e]
at org.apache.pinot.core.startree.v2.builder.MultipleTreesBuilder.build(MultipleTreesBuilder.java:130) ~[pinot-all-0.5.0-2020-08-13-SNAPSHOT-jar-with-dependencies.jar:0.5.0-2020-08-13-SNAPSHOT-5ac67eb1a1210cb6c0c76d81d138cac29d0d3d4e]
at org.apache.pinot.core.segment.creator.impl.SegmentIndexCreationDriverImpl.buildStarTreeV2IfNecessary(SegmentIndexCreationDriverImpl.java:298) ~[pinot-all-0.5.0-2020-08-13-SNAPSHOT-jar-with-dependencies.jar:0.5.0-2020-08-13-SNAPSHOT-5ac67eb1a1210cb6c0c76d81d138cac29d0d3d4e]
at org.apache.pinot.core.segment.creator.impl.SegmentIndexCreationDriverImpl.handlePostCreation(SegmentIndexCreationDriverImpl.java:263) ~[pinot-all-0.5.0-2020-08-13-SNAPSHOT-jar-with-dependencies.jar:0.5.0-2020-08-13-SNAPSHOT-5ac67eb1a1210cb6c0c76d81d138cac29d0d3d4e]
at org.apache.pinot.core.segment.creator.impl.SegmentIndexCreationDriverImpl.build(SegmentIndexCreationDriverImpl.java:223) ~[pinot-all-0.5.0-2020-08-13-SNAPSHOT-jar-with-dependencies.jar:0.5.0-2020-08-13-SNAPSHOT-5ac67eb1a1210cb6c0c76d81d138cac29d0d3d4e]
at org.apache.pinot.tools.admin.command.CreateSegmentCommand.lambda$execute$0(CreateSegmentCommand.java:239) ~[pinot-all-0.5.0-2020-08-13-SNAPSHOT-jar-with-dependencies.jar:0.5.0-2020-08-13-SNAPSHOT-5ac67eb1a1210cb6c0c76d81d138cac29d0d3d4e]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) ~[?:?]
at java.lang.Thread.run(Thread.java:832) ~[?:?]
```
Contributor guide
Research direction
Reproduce the failure with the schema and table configurations using pinot-admin.sh CreateSegment. Start with CreateSegmentCommand.java and the stack-trace path through ImmutableDataSource.java and BaseSingleTreeBuilder.java; done means an invalid star-tree column no longer produces an unexplained NullPointerException.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100