apache / apache/pinot

create real-time segment error

Open
#6,624 7 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 3h
Merged PRs (30d)
195

Description

error log :
INFO [SegmentDictionaryCreator] [wme_pinot__12__0__20210301T1252Z] Created dictionary for LONG column: receive_video_minutes with cardinality: 2, range: 0 to 1
2021/03/01 12:53:48.100 INFO [BaseSingleTreeBuilder] [wme_pinot__3__0__20210301T1252Z] Starting building star-tree with config: StarTreeV2BuilderConfig[splitOrder=[date_time, webex_site_id, network_type],skipStarNodeCreation=[],functionColumnPairs=[sum__bad_video_minutes, sum__receive_bad_video_minutes, sum__bad_audio_minutes, sum__send_video_minutes, sum__video_minutes, sum__audio_minutes, sum__send_bad_video_minutes, sum__receive_video_minutes],maxLeafRecords=10000]
2021/03/01 12:53:48.100 INFO [SegmentDictionaryCreator] [wme_pinot__12__0__20210301T1252Z] Using fixed bytes value dictionary for column: server_region, size: 242
2021/03/01 12:53:48.101 INFO [SegmentDictionaryCreator] [wme_pinot__12__0__20210301T1252Z] Created dictionary for STRING column: server_region with cardinality: 11, max length in bytes: 22, range: Amsterdam, Netherlands to unknown
2021/03/01 12:53:48.102 INFO [ImmutableSegmentImpl] [wme_pinot__3__0__20210301T1252Z] Trying to destroy segment : wme_pinot__3__0__20210301T1252Z
2021/03/01 12:53:48.102 INFO [SegmentDictionaryCreator] [wme_pinot__12__0__20210301T1252Z] Created dictionary for LONG column: receive_bad_video_minutes with cardinality: 2, range: 0 to 1
2021/03/01 12:53:48.102 ERROR [LLRealtimeSegmentDataManager_wme_pinot__3__0__20210301T1252Z] [wme_pinot__3__0__20210301T1252Z] Could not build segment
java.lang.UnsupportedOperationException: null
at org.apache.pinot.core.segment.index.readers.ForwardIndexReader.getDictId(ForwardIndexReader.java:71) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
at org.apache.pinot.core.data.readers.PinotSegmentColumnReader.getDictId(PinotSegmentColumnReader.java:54) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
at org.apache.pinot.core.startree.v2.builder.BaseSingleTreeBuilder.getSegmentRecordDimensions(BaseSingleTreeBuilder.java:224) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
at org.apache.pinot.core.startree.v2.builder.OffHeapSingleTreeBuilder.sortAndAggregateSegmentRecords(OffHeapSingleTreeBuilder.java:213) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
at org.apache.pinot.core.startree.v2.builder.BaseSingleTreeBuilder.build(BaseSingleTreeBuilder.java:304) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
at org.apache.pinot.core.startree.v2.builder.OffHeapSingleTreeBuilder.build(OffHeapSingleTreeBuilder.java:43) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
at org.apache.pinot.core.startree.v2.builder.MultipleTreesBuilder.build(MultipleTreesBuilder.java:142) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]


schema config :
{
"schemaName":"xxx",
"dimensionFieldSpecs":[
{
"name":"d1",
"dataType":"STRING",
"singleValueField":false
},
{
"name":"d2",
"dataType":"STRING",
"singleValueField":false
},
{
"name":"d3",
"dataType":"STRING",
"singleValueField":false
},
{
"name":"d4",
"dataType":"STRING",
"singleValueField":false
},
{
"name":"d5",
"dataType":"STRING",
"singleValueField":false
}
],
"metricFieldSpecs":[
{
"name":"m1",
"dataType":"LONG",
"defaultNullValue":0
},
{
"name":"m2",
"dataType":"LONG",
"defaultNullValue":0
},
{
"name":"m3",
"dataType":"LONG",
"defaultNullValue":0
},
{
"name":"m4",
"dataType":"LONG",
"defaultNullValue":0
},
{
"name":"m5",
"dataType":"LONG",
"defaultNullValue":0
},
{
"name":"m6",
"dataType":"LONG",
"defaultNullValue":0
},
{
"name":"m7",
"dataType":"LONG",
"defaultNullValue":0
},
{
"name":"m8",
"dataType":"LONG",
"defaultNullValue":0
}
],
"dateTimeFieldSpecs":[
{
"name":"date_time",
"dataType":"STRING",
"format":"1:DAYS:SIMPLE_DATE_FORMAT:yyyy-MM-dd",
"granularity":"1:DAYS"
}
]
}

table config :
{
"tableName":"tableName",
"tableType":"REALTIME",
"segmentsConfig":{
"timeColumnName":"date_time",
"timeType":"DAYS",
"retentionTimeUnit":"DAYS",
"retentionTimeValue":"5",
"segmentPushType":"APPEND",
"segmentAssignmentStrategy":"BalanceNumSegmentAssignmentStrategy",
"schemaName":"xxx",
"replication":"1",
"replicasPerPartition":"1"
},
"tenants":{

},
"tableIndexConfig":{
"loadMode":"MMAP",
"enableDefaultStarTree":true,
"streamConfigs":{
"streamType":"kafka",
"stream.kafka.consumer.type":"simple",
"stream.kafka.topic.name":"local_mqa_telemetry_wmequality_report",
"stream.kafka.decoder.class.name":"org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder",
"stream.kafka.consumer.factory.class.name":"org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
"stream.kafka.broker.list":"10.250.84.160:9092",
"realtime.segment.flush.threshold.time":"3600000",
"realtime.segment.flush.threshold.size":"50000",
"stream.kafka.consumer.prop.auto.offset.reset":"largest"
},
"starTreeIndexConfigs":[
{
"dimensionsSplitOrder":[
"d1",
"d2",
"d3",
"d4",
"d5"
],
"skipStarNodeCreationForDimensions":[

],
"functionColumnPairs":[
"SUM__m1",
"SUM__m2",
"SUM__m3",
"SUM__m4",
"SUM__m5",
"SUM__m6",
"SUM__m7",
"SUM__m8"
]
}
]
},
"metadata":{
"customConfigs":{

}
}
}

Contributor guide

Open the contributing guide

Research direction

Start at org.apache.pinot.core.segment.index.readers.ForwardIndexReader.getDictId and trace its call through PinotSegmentColumnReader, BaseSingleTreeBuilder, and OffHeapSingleTreeBuilder using the supplied realtime schema and table configuration. Reproduce the star-tree build failure and determine why getDictId is invoked for this segment; done means the realtime segment builds successfully without the UnsupportedOperationException.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.