apache / apache/pinot

UI - Reporting invalid configurations during table creation

Open
#6,850 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

Bug found in pinot v0.7.0

When creating a table definition in the UI, it will not allow definitions with `upsertConfig` due to reporting that such configurations are invalid.

A workaround for this is to directly use the swagger API. It seems the UI is not using the same backend service for configuration validation.

An example configuration could be:
```
{
"REALTIME": {
"tableName": "Table_REALTIME",
"tableType": "REALTIME",
"segmentsConfig": {
"timeColumnName": "timestamp",
"replication": "1",
"replicasPerPartition": "1",
"schemaName": "Table"
},
"tenants": {
"broker": "DefaultTenant",
"server": "DefaultTenant",
"tagOverrideConfig": {}
},
"tableIndexConfig": {
"invertedIndexColumns": [],
"rangeIndexColumns": [],
"varLengthDictionaryColumns": [],
"enableDefaultStarTree": false,
"enableDynamicStarTreeCreation": false,
"aggregateMetrics": false,
"nullHandlingEnabled": false,
"autoGeneratedInvertedIndex": false,
"createInvertedIndexDuringSegmentGeneration": false,
"bloomFilterColumns": [],
"loadMode": "MMAP",
"streamConfigs": {
"streamType": "kafka",
"stream.kafka.topic.name": "data.Table",
"stream.kafka.broker.list": "kafka.dc-kafka.svc.cluster.local:9092",
"stream.kafka.consumer.type": "lowlevel",
"stream.kafka.consumer.prop.auto.offset.reset": "smallest",
"stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
"stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder",
"realtime.segment.flush.threshold.rows": "0",
"realtime.segment.flush.threshold.time": "24h",
"realtime.segment.flush.segment.size": "100M"
},
"noDictionaryColumns": [],
"onHeapDictionaryColumns": [],
"sortedColumn": []
},
"metadata": {},
"quota": {},
"routing": {},
"query": {},
"upsertConfig": {
"mode": "FULL"
},
"ingestionConfig": {},
"isDimTable": false
}
}
```

With a schema definition:
```
{
"schemaName": "Table",
"dimensionFieldSpecs": [
{
"name": "executionId",
"dataType": "STRING"
},
{
"name": "jobId",
"dataType": "STRING"
},
{
"name": "jobMemberId",
"dataType": "STRING"
},
{
"name": "projectId",
"dataType": "STRING"
}
],
"dateTimeFieldSpecs": [
{
"name": "timestamp",
"dataType": "LONG",
"format": "1:MILLISECONDS:EPOCH",
"granularity": "1:MILLISECONDS"
}
],
"primaryKeyColumns": [
"executionId"
]
}
```

Contributor guide

Open the contributing guide

Research direction

Reproduce the problem in the table-creation UI using the supplied table and schema definitions, then compare its validation behavior with the Swagger API workaround. The work is done when a table definition containing upsertConfig is accepted consistently by the UI and backend.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
frontend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.