apache / apache/pulsar

python function support multiple input typesClassNames

Open
#21,539 2 comments 0 reactions 0 assignees View on GitHub
type/enhancement
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### Motivation

as it is possible in the yaml to define multiple input topics it should be possible as well to reference the typeClassName on the inputspecs level and not only in the input section

```
"source": {
"typeClassName": "schema-test.EasySchema",
"inputSpecs": {
"tenant/test-schema/schema.v1": {
"serdeClassName": "schema-test.EasySchemaSerDe",
"schemaType": "avro"
},
"tenant/test-schema/schema.v2": {
"serdeClassName": "schema-test.EasySchemaSerDe",
"schemaType": "avro"
}
},
"subscriptionPosition": "EARLIEST"
}
```

Else it isn't possible to define different class references for different topics (if the Class and SerDe is different)

in [get_schema()](https://github.com/apache/pulsar/blob/master/pulsar-functions/instance/src/main/python/python_instance.py#L195) it should have the possibility to reference consumer_conf.typeClassName

so final spec would look like
```
"source": {
"inputSpecs": {
"tenant/test-schema/schema.v1": {
"serdeClassName": "schema-test.EasySchemaSerDe",
"schemaType": "avro",
"typeClassName": "schema-test.EasySchema"
},
"tenant/test-schema/schema.v2": {
"serdeClassName": "schema-test.EasySchema2SerDe",
"schemaType": "avro",
"typeClassName": "schema-test.EasySchema2",
}
},
"subscriptionPosition": "EARLIEST"
}
```

### Solution

_No response_

### Alternatives

_No response_

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start with pulsar-functions/instance/src/main/python/python_instance.py and its get_schema() entry point. Trace how consumer_conf.typeClassName is read today and compare it with the inputSpecs entries in the issue. Done means each input topic can use its own typeClassName and serde configuration, with coverage for the separate topic definitions.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, python
Domain
backend, distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.