All table columns have default values
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 195
Description
I have the following avro schema definition:
```
{
"type": "record",
"name": "QueryWithKpis",
"namespace": "foo.bar.v1",
"fields": [
{
"name": "query",
"type": {
"type": "record",
"name": "MappedQuery",
"namespace": "foo.bar.v1",
"fields": [
{
"name": "userQuery",
"type": {
"type": "string",
"avro.java.string": "String"
}
},
{
"name": "masterQuery",
"type": [
"null",
{
"type": "string",
"avro.java.string": "String"
}
]
},
{
"name": "mappingTime",
"type": "long",
"order": "ignore"
}
]
}
},
{
"name": "kpis",
"type": {
"type": "map",
"values": "double",
"avro.java.string": "String"
}
}
]
}
```
and the following schema definition:
```
{
"schemaName": "kpis",
"dimensionFieldSpecs": [
{
"name": "query.userQuery",
"dataType": "STRING"
},
{
"name": "query.masterQuery",
"dataType": "STRING"
}
],
"metricFieldSpecs": [
{
"name": "kpis.clicks",
"dataType": "INT"
},
{
"name": "kpis.checkout",
"dataType": "INT"
},
{
"name": "kpis.cart",
"dataType": "INT"
}
],
"dateTimeFieldSpecs": [
{
"name": "timestamp",
"dataType": "LONG",
"format": "1:SECONDS:EPOCH",
"granularity": "15:MINUTES"
}
]
}
```
After running a batch ingestion job, all database rows contain the default value of the field's dataType. I tried ingesting the data with complex type config and without:
```
"complexTypeConfig": {
"fieldsToUnnest": [
"query",
"kpis",
]
}
```
I've created a small Gist and checked the output of https://github.com/apache/pinot/blob/master/pinot-plugins/pinot-input-format/pinot-avro-base/src/main/java/org/apache/pinot/plugin/inputformat/avro/AvroRecordReader.java
```
public static void main(String[] args) {
try (
DataFileStream dataFileStream = new DataFileStream<>(
new FileInputStream(Paths.get("/someKpis.avro").toFile()), new GenericDatumReader<>())
) {
while (dataFileStream.hasNext()) {
GenericRecord from = dataFileStream.next();
List fields = from.getSchema().getFields();
Object test = from.get("query.userQuery");
System.out.println("query:" + test);
for (Schema.Field field : fields) {
String fieldName = field.name();
Object value = from.get(fieldName);
if (value != null) {
value = convert(value);
}
System.out.println("fieldName: " + fieldName + " - value: " + value);
}
}
}
catch (
IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
```
Which prints out the following:
```
fieldName: query - value: {masterQuery=null, mappingTime=-123213 userQuery=dekoleiter}
fieldName: kpis - value: {clicks=3.0, checkout=0.0, cart=1.0}
...
```
Any ideas?
Contributor guide
Research direction
Start by reading pinot-plugins/pinot-input-format/pinot-avro-base/src/main/java/org/apache/pinot/plugin/inputformat/avro/AvroRecordReader.java and reproduce the batch ingestion with the provided Avro and Pinot schemas, both with and without complexTypeConfig. Trace how nested query and kpis values are read and mapped; done means ingested rows contain the source values instead of data-type defaults.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, database
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100