apache / apache/pinot

All table columns have default values

Open
#7,262 20 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 3h
Merged PRs (30d)
195

Description

I have the following avro schema definition:
```
{
"type": "record",
"name": "QueryWithKpis",
"namespace": "foo.bar.v1",
"fields": [
{
"name": "query",
"type": {
"type": "record",
"name": "MappedQuery",
"namespace": "foo.bar.v1",
"fields": [
{
"name": "userQuery",
"type": {
"type": "string",
"avro.java.string": "String"
}
},
{
"name": "masterQuery",
"type": [
"null",
{
"type": "string",
"avro.java.string": "String"
}
]
},
{
"name": "mappingTime",
"type": "long",
"order": "ignore"
}
]
}
},
{
"name": "kpis",
"type": {
"type": "map",
"values": "double",
"avro.java.string": "String"
}
}
]
}
```
and the following schema definition:
```
{
"schemaName": "kpis",
"dimensionFieldSpecs": [
{
"name": "query.userQuery",
"dataType": "STRING"
},
{
"name": "query.masterQuery",
"dataType": "STRING"
}
],
"metricFieldSpecs": [
{
"name": "kpis.clicks",
"dataType": "INT"
},
{
"name": "kpis.checkout",
"dataType": "INT"
},
{
"name": "kpis.cart",
"dataType": "INT"
}
],
"dateTimeFieldSpecs": [
{
"name": "timestamp",
"dataType": "LONG",
"format": "1:SECONDS:EPOCH",
"granularity": "15:MINUTES"
}
]
}
```
After running a batch ingestion job, all database rows contain the default value of the field's dataType. I tried ingesting the data with complex type config and without:
```
"complexTypeConfig": {
"fieldsToUnnest": [
"query",
"kpis",
]
}
```
I've created a small Gist and checked the output of https://github.com/apache/pinot/blob/master/pinot-plugins/pinot-input-format/pinot-avro-base/src/main/java/org/apache/pinot/plugin/inputformat/avro/AvroRecordReader.java
```
public static void main(String[] args) {
try (
DataFileStream dataFileStream = new DataFileStream<>(
new FileInputStream(Paths.get("/someKpis.avro").toFile()), new GenericDatumReader<>())
) {
while (dataFileStream.hasNext()) {
GenericRecord from = dataFileStream.next();
List fields = from.getSchema().getFields();

Object test = from.get("query.userQuery");
System.out.println("query:" + test);
for (Schema.Field field : fields) {
String fieldName = field.name();
Object value = from.get(fieldName);
if (value != null) {
value = convert(value);
}
System.out.println("fieldName: " + fieldName + " - value: " + value);
}
}
}
catch (

IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
```
Which prints out the following:
```
fieldName: query - value: {masterQuery=null, mappingTime=-123213 userQuery=dekoleiter}
fieldName: kpis - value: {clicks=3.0, checkout=0.0, cart=1.0}
...
```
Any ideas?

Contributor guide

Open the contributing guide

Research direction

Start by reading pinot-plugins/pinot-input-format/pinot-avro-base/src/main/java/org/apache/pinot/plugin/inputformat/avro/AvroRecordReader.java and reproduce the batch ingestion with the provided Avro and Pinot schemas, both with and without complexTypeConfig. Trace how nested query and kpis values are read and mapped; done means ingested rows contain the source values instead of data-type defaults.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, database
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.