apache / apache/pinot

Json ingestion failing for some array types

Open
#8,635 3 comments 0 reactions 1 assignee Claimed by @KKcorps View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

I have found that I can ingest using JSON all types as multi-valued dimension columns with the exception of BOOLEAN, TIMESTAMP, and BYTES. I believe that JSON_ARRAY isn't a valid type, but I wasn't sure about BYTES_ARRAY. If the muli-valued versions of those types are removed from the schema and data files below, ingestion succeeds and I can inspect the table in the pinot browser. If anyone is about and can shed some light, I would be very appreciative.

# Json Ingestion

For BYTES_ARRAY I get:
```
java.lang.UnsupportedOperationException: Unsupported data type : BYTES
```

For BOOLEAN_ARRAY I get:
```
java.lang.ClassCastException: class [Z cannot be cast to class java.lang.Integer ([Z and java.lang.Integer are in module java.base of loader 'bootstrap')
```

For TIMESTAMP_ARRAY I get:
```
java.lang.ClassCastException: class java.sql.Timestamp cannot be cast to class java.lang.Long (java.sql.Timestamp is in module java.sql of loader 'platform'; java.lang.Long is in module java.base of loader 'bootstrap')
```

Here is my cluster:
```yaml
version: "3"

services:
pinot-zookeeper:
image: apachepinot/pinot:release-0.10.0
hostname: pinot-zookeeper
container_name: "pinot-client-rust-pinot-zookeeper"
ports:
- "2181:2181"
command: StartZookeeper

pinot-controller:
image: apachepinot/pinot:release-0.10.0
hostname: pinot-controller
container_name: "pinot-client-rust-pinot-controller"
volumes:
- ./db:/db
ports:
- "9000:9000"
command: StartController -zkAddress pinot-zookeeper:2181
depends_on:
- pinot-zookeeper

pinot-broker:
image: apachepinot/pinot:release-0.10.0
hostname: pinot-broker
container_name: "pinot-client-rust-pinot-broker"
volumes:
- ./db:/db
ports:
- "8099:8099"
command: StartBroker -zkAddress pinot-zookeeper:2181
restart: unless-stopped
depends_on:
- pinot-zookeeper
- pinot-controller

pinot-server:
image: apachepinot/pinot:release-0.10.0
hostname: pinot-server
container_name: "pinot-client-rust-pinot-server"
volumes:
- ./db:/db
ports:
- "8098:8098"
command: StartServer -zkAddress pinot-zookeeper:2181
depends_on:
- pinot-zookeeper
- pinot-controller

# cargo will try to redownload packages @ docker-compose up so store them here.
volumes:
pgdata: {}

```

Here is my schema:
```json
{
"schemaName": "scoreSheet",
"dimensionFieldSpecs": [
{
"name": "handle",
"dataType": "STRING"
},
{
"name": "names",
"dataType": "STRING",
"singleValueField": false
},
{
"name": "age",
"dataType": "INT"
},
{
"name": "gameIds",
"dataType": "INT",
"singleValueField": false
},
{
"name": "hasPlayed",
"dataType": "BOOLEAN"
},
{
"name": "gamesWon",
"dataType": "BOOLEAN",
"singleValueField": false
},
{
"name": "dateOfBirth",
"dataType": "TIMESTAMP"
},
{
"name": "datesPlayed",
"dataType": "TIMESTAMP",
"singleValueField": false
},
{
"name": "scores",
"dataType": "LONG",
"singleValueField": false
},
{
"name": "handicapAdjustedScores",
"dataType": "FLOAT",
"singleValueField": false
},
{
"name": "handicapAdjustedScores_highPrecision",
"dataType": "FLOAT",
"singleValueField": false
},
{
"name": "extra",
"dataType": "JSON"
},
{
"name": "raw",
"dataType": "BYTES"
},
{
"name": "rawArray",
"dataType": "BYTES",
"singleValueField": false
}
],
"metricFieldSpecs": [
{
"name": "totalScore",
"dataType": "LONG"
},
{
"name": "avgScore",
"dataType": "FLOAT"
},
{
"name": "avgScore_highPrecision",
"dataType": "DOUBLE"
}
],
"dateTimeFieldSpecs": [
{
"name": "dateOfFirstGame",
"dataType": "LONG",
"format": "1:MILLISECONDS:EPOCH",
"granularity": "1:MILLISECONDS"
}
]
}

```
Here is my table:
```json
{
"tableName": "scoreSheet",
"tableType": "OFFLINE",
"segmentsConfig": {
"replication": 1
},
"tenants": {
"broker":"DefaultTenant",
"server":"DefaultTenant"
},
"tableIndexConfig": {
"loadMode": "MMAP"
},
"ingestionConfig": {
"batchIngestionConfig": {
"segmentIngestionType": "APPEND",
"segmentIngestionFrequency": "DAILY"
}
},
"metadata": {}
}

```

Here is my ingestion job:
```yaml
executionFrameworkSpec:
name: 'standalone'
segmentGenerationJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner'
segmentTarPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner'
jobType: SegmentCreationAndTarPush
inputDirURI: '/db/score_sheet'
# includeFileNamePattern: 'glob:**/data.csv'
includeFileNamePattern: 'glob:**/data.json'
outputDirURI: '/opt/pinot/data/score_sheet'
overwriteOutput: true
pinotFSSpecs:
- scheme: file
className: org.apache.pinot.spi.filesystem.LocalPinotFS
recordReaderSpec:
# dataFormat: 'csv'
dataFormat: 'json'
# className: 'org.apache.pinot.plugin.inputformat.csv.CSVRecordReader'
className: 'org.apache.pinot.plugin.inputformat.json.JSONRecordReader'
# configClassName: 'org.apache.pinot.plugin.inputformat.csv.CSVRecordReaderConfig'
tableSpec:
tableName: 'scoreSheet'
pinotClusterSpecs:
- controllerURI: 'http://localhost:9000'

```

Here is my data:
```json
[
{
"names": ["James", "Smith"],
"gameIds": [1, 2, 3],
"datesPlayed": ["2020-01-01 10:45:28", "2020-02-01 10:45:28", "2020-03-01 10:45:28"],
"gamesWon": [true, false, true],
"scores": [3, 6, 2],
"handicapAdjustedScores": [2.1, 4.9, 3.2],
"handicapAdjustedScores_highPrecision": [2.15, 4.99, 3.21],
"rawArray": ["cd", "ef"],
"handle": "Gladiator",
"age": 10,
"totalScore": 11,
"avgScore": 3.6,
"avgScore_highPrecision": 3.66,
"hasPlayed": true,
"dateOfBirth": "2011-01-01 00:00:00",
"dateOfFirstGame": 1577875528000,
"extra": "{\"a\": \"b\"}",
"raw": "ab"
},
{
"names": ["Giles", "Richie"],
"gameIds": [],
"datesPlayed":[] ,
"gamesWon": [],
"scores": [],
"handicapAdjustedScores": [],
"handicapAdjustedScores_highPrecision": [],
"rawArray": [],
"handle": "Thrumbar",
"age": 30,
"totalScore": 0,
"avgScore": 0,
"avgScore_highPrecision": 0,
"hasPlayed": false,
"dateOfBirth": 662688000000,
"dateOfFirstGame": 1420070400001,
"extra": {},
"raw": ""
}
]
```

# CSV Ingestion

Using the following ingestion job, schema, and csv file, an INT was ingested to an INT multi-value column:

Ingestion job:

```yaml
executionFrameworkSpec:
name: 'standalone'
segmentGenerationJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner'
segmentTarPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner'
jobType: SegmentCreationAndTarPush
inputDirURI: '/db/score_sheet'
includeFileNamePattern: 'glob:**/data.csv'
# includeFileNamePattern: 'glob:**/data.json'
outputDirURI: '/opt/pinot/data/score_sheet'
overwriteOutput: true
pinotFSSpecs:
- scheme: file
className: org.apache.pinot.spi.filesystem.LocalPinotFS
recordReaderSpec:
dataFormat: 'csv'
# dataFormat: 'json'
className: 'org.apache.pinot.plugin.inputformat.csv.CSVRecordReader'
# className: 'org.apache.pinot.plugin.inputformat.json.JSONRecordReader'
configClassName: 'org.apache.pinot.plugin.inputformat.csv.CSVRecordReaderConfig'
tableSpec:
tableName: 'scoreSheet'
pinotClusterSpecs:
- controllerURI: 'http://localhost:9000'
```

Schema:

```json
{
"schemaName": "scoreSheet",
"dimensionFieldSpecs": [
{
"name": "gamesWon",
"dataType": "INT",
"singleValueField": false
}
],
"metricFieldSpecs": [
],
"dateTimeFieldSpecs": [
]
}
```
CSV:

```csv
gamesWon
1
```
However, the aforementioned error from json ingestion presents when tried for booleans:

Error:

```
2022/05/05 08:44:40.594 ERROR [SegmentGenerationJobRunner] [pool-2-thread-1] Failed to generate Pinot segment for file - file:/db/score_sheet/data.json
java.lang.ClassCastException: class [Z cannot be cast to class java.lang.Integer ([Z and java.lang.Integer are in module java.base of loader 'bootstrap')
```
Schema:

```json
{
"schemaName": "scoreSheet",
"dimensionFieldSpecs": [
{
"name": "gamesWon",
"dataType": "BOOLEAN",
"singleValueField": false
}
],
"metricFieldSpecs": [
],
"dateTimeFieldSpecs": [
]
}
```
CSV:

```csv
gamesWon
true
```
and

```csv
gamesWon
1
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.