apache / apache/pulsar

Pulsar Java client's RecordSchemaBuilder doesn't provide any APIs to add nested records or Arrays for Avro and JSON type

Open
#20,275 2 comments 0 reactions 0 assignees View on GitHub
Stale type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### Version

Java Client(2.11.0)-->broker(2.11.0)
Broker instance OS: Amazon Linux 2023
Java App running OS : Windows 10

### Minimal reproduce step

- Using the RecordSchemaBuilder API I couldn't add any nested elements.
- Refer to the following code snippet which has nested records and array elements in the Schema definition.
- Using the record schema builder I couldn't find a way to add nested records or arrays.
- This applicable for both Avro as well as JSON. Hence request you to provide recommendations on how to achieve such use cases.

```
String avroSchemaDef= "{
"namespace": "my.com.ns",
"name": "myrecord",
"type": "record",
"fields": [{
"name": "uid",
"type": "int"
},
{
"name": "Product",
"type": "string"
},
{
"name": "options",
"type": {
"type": "array",
"items": {
"type": "record",
"name": "lvl2_record",
"fields": [{
"name": "item1_lvl2",
"type": "string"
},
{"name": "additional",
"type": {"type": "map", "values": "string"}},
{
"name": "item2_lvl2",
"type": {
"type": "array",
"items": {
"type": "record",
"name": "lvl3_record",
"fields": [{
"name": "item1_lvl3",
"type": "string"
},
{
"name": "item2_lvl3",
"type": ["string", "null"]
}, {
"name": "item_status",
"type": {
"name": "item_status",
"type": "enum",
"symbols": ["AVAILABLE", "OUT_OF_STOCK", "ONLY_FEW_LEFT"]
},
"default": "AVAILABLE"
}
]
}
}
}
]
}
}
}
]
}";
JSONObject schemaDefObj = new JSONObject(avroSchemaDef);
JSONArray fArray = schemaDefObj.getJSONArray("fields");
RecordSchemaBuilder recordSchemaBuilder = SchemaBuilder.record(schemaName);
for(int i=0;i

Contributor guide

Open the contributing guide

Research direction

Start with the RecordSchemaBuilder and SchemaBuilder.record entry points mentioned in the issue, then trace how SchemaType and build(SchemaType) represent fields for Avro and JSON. Done should mean the builder offers a documented way to represent nested records and arrays for both schema types, with coverage for the nested example described here.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, json
Domain
backend-api-design
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.