apache / apache/pulsar

Schema.json: Type not supported: object

Open
#14,932 5 comments 0 reactions 0 assignees View on GitHub
lifecycle/stale Stale type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### **Describe the bug**

We use schema to send messages, and the creation fails during the process of creating the producer. The error log is: caused by org.apache.avro.SchemaParseException: Type not supported: object. We suspect that there is a problem with Schema.json(). When we change a topic, the log no longer reports an error.

Pulsar: v2.8.2

### **To Reproduce**

Steps to reproduce the behavior:

1. Schema class

```java
@Data
public class StudentUploadQuestionResultMsg {
private String studentId;
private String classId;
private String schoolId;
private Integer subjectId; // SubjectEnum.pushValue
private Long uploadTime;
private List uploadQuestions;
private String dataSource;
@Data
public static class UploadQuestion {
private String answerId;
private String questionId;
private Integer answerResult;
private List tag;
}
}
```

2. Producer Code

```java
producer = client.newProducer(Schema.JSON(StudentUploadQuestionResultMsg.class)).topic(studentUploadQuestionTopic).create();
```

3. see error

```
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.apache.pulsar.client.api.Producer]: Factory method 'studentUploadQuestionProducer' threw exception; nested exception is org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.ExecutionException: org.apache.pulsar.client.api.PulsarClientException: org.apache.avro.SchemaParseException: Type not supported: object caused by org.apache.avro.SchemaParseException: Type not supported: object
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:185)
at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:622)
... 66 common frames omitted
Caused by: org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.ExecutionException: org.apache.pulsar.client.api.PulsarClientException: org.apache.avro.SchemaParseException: Type not supported: object caused by org.apache.avro.SchemaParseException: Type not supported: object
at org.apache.pulsar.client.api.PulsarClientException.unwrap(PulsarClientException.java:1027)
at org.apache.pulsar.client.impl.ProducerBuilderImpl.create(ProducerBuilderImpl.java:95)
at com.youdao.correctionReport.config.mq.PulsarConfig.studentUploadQuestionProducer(PulsarConfig.java:80)
at com.youdao.correctionReport.config.mq.PulsarConfig$$EnhancerBySpringCGLIB$$34e7bea7.CGLIB$studentUploadQuestionProducer$11()
at com.youdao.correctionReport.config.mq.PulsarConfig$$EnhancerBySpringCGLIB$$34e7bea7$$FastClassBySpringCGLIB$$60fba01a.invoke()
at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:244)
at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:363)
at com.youdao.correctionReport.config.mq.PulsarConfig$$EnhancerBySpringCGLIB$$34e7bea7.studentUploadQuestionProducer()
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154)
... 67 common frames omitted
Caused by: java.util.concurrent.ExecutionException: org.apache.pulsar.client.api.PulsarClientException: org.apache.avro.SchemaParseException: Type not supported: object caused by org.apache.avro.SchemaParseException: Type not supported: object
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
at org.apache.pulsar.client.impl.ProducerBuilderImpl.create(ProducerBuilderImpl.java:93)
... 78 common frames omitted
Caused by: org.apache.pulsar.client.api.PulsarClientException: org.apache.avro.SchemaParseException: Type not supported: object caused by org.apache.avro.SchemaParseException: Type not supported: object
```

### **Additional context**

Two things strike us as strange:

1. We have been producing data to this topic for a while. The behavior is normal. This error is originated when we updated a the schema class.

```java
// old schema class
@Data
public class StudentUploadQuestionResultMsg {
private String studentId;
private String classId;
private String schoolId;
private Integer commonSubjectId;
private Long uploadTime;
private List uploadQuestions;

@Data
public static class UploadQuestion {
private String answerId;
private String questionId;
private Integer answerResult;
}
}
```

2. We used restapi to check the schema infomation. We found a strange schema which VERSION is 4. The autoupdateschema is true and Schema compatibility check strategy is FULL.

```json
// strange schema. It has no name and type is object.
{
"type":"object",
"id":"urn:jsonschema:com:youdao:correctionReport:answer:mq:StudentUploadQuestionResultMsg",
"properties":{
"studentId":{
"type":"string"
},
"classId":{
"type":"string"
},
"schoolId":{
"type":"string"
},
"commonSubjectId":{
"type":"integer"
},
"uploadTime":{
"type":"integer"
},
"uploadQuestions":{
"type":"array",
"items":{
"type":"object",
"id":"urn:jsonschema:com:youdao:correctionReport:answer:mq:StudentUploadQuestionResultMsg:UploadQuestion",
"properties":{
"answerId":{
"type":"string"
},
"questionId":{
"type":"string"
},
"answerResult":{
"type":"integer"
}
}
}
}
}
}

// nomal schema
{
"type":"record",
"name":"StudentUploadQuestionResultMsg",
"namespace":"com.youdao.correctionReport.answer.mq",
"fields":[
{
"name":"classId",
"type":[
"null",
"string"
],
"default":null
},
{
"name":"commonSubjectId",
"type":[
"null",
"int"
],
"default":null
},
{
"name":"schoolId",
"type":[
"null",
"string"
],
"default":null
},
{
"name":"studentId",
"type":[
"null",
"string"
],
"default":null
},
{
"name":"uploadQuestions",
"type":[
"null",
{
"type":"array",
"items":{
"type":"record",
"name":"UploadQuestion",
"namespace":"com.youdao.correctionReport.answer.mq.StudentUploadQuestionResultMsg",
"fields":[
{
"name":"answerId",
"type":[
"null",
"string"
],
"default":null
},
{
"name":"answerResult",
"type":[
"null",
"int"
],
"default":null
},
{
"name":"questionId",
"type":[
"null",
"string"
],
"default":null
}
]
},
"java-class":"java.util.List"
}
],
"default":null
},
{
"name":"uploadTime",
"type":[
"null",
"long"
],
"default":null
}
]
}
```

Contributor guide

Open the contributing guide

Research direction

Start with Schema.JSON generation and the schema registration path reached by ProducerBuilderImpl.create; compare the reported object schema with the normal Avro schema and the model changes that preceded it. Reproduce the producer creation failure with the supplied StudentUploadQuestionResultMsg classes and determine why the object schema is registered or consumed; done means the cause and a verified correction or documented limitation are established.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.