[Bug] Sink inputSpecs schemaProperties is silently dropped by SinkConfigUtils in both conversion directions
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 160
Description
**Search before asking**
- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### Version
Reproduced on `master` (`e4d106f6ee`). The same omission is present on the 4.0.x line.
### Minimal reproduce step
Create a sink whose `inputSpecs` entry carries `schemaProperties`:
```json
{
"tenant": "public",
"namespace": "default",
"name": "test-sink",
"inputSpecs": {
"persistent://public/default/in": {
"schemaType": "avro",
"schemaProperties": { "__alwaysAllowNull": "true" },
"consumerProperties": { "application": "billing" }
}
}
}
```
Then read it back:
```
pulsar-admin sinks get --tenant public --namespace default --name test-sink
```
### What did you expect to see?
`schemaProperties` preserved on the round trip, the same way `consumerProperties` is, and the same way it already works for Pulsar Functions.
### What did you see instead?
`schemaProperties` is silently dropped. The API accepts the value, but it never reaches `ConsumerSpec` and is absent from the response. There is no error or warning, so the caller has no signal that the setting was discarded.
### Anything else?
The proto field exists — `Function.proto:111` defines `map schemaProperties = 5` on `ConsumerSpec` — and `ConsumerConfig.schemaProperties` exists on the model. Only the **sink** conversion path fails to use them, in both directions:
- [`SinkConfigUtils.convert()`](https://github.com/apache/pulsar/blob/master/pulsar-functions/utils/src/main/java/org/apache/pulsar/functions/utils/SinkConfigUtils.java#L148-L171) builds each `ConsumerSpec` from the `inputSpecs` entry and copies `schemaType`/`serdeClassName`, `receiverQueueSize`, `cryptoSpec`, `messagePayloadProcessorSpec`, `consumerProperties` and `poolMessages` — but never `schemaProperties`.
- [`SinkConfigUtils.convertFromDetails()`](https://github.com/apache/pulsar/blob/master/pulsar-functions/utils/src/main/java/org/apache/pulsar/functions/utils/SinkConfigUtils.java#L290-L313) restores the same set on the way back, and likewise never restores `schemaProperties`.
`FunctionConfigUtils` handles the field correctly in both directions (`putSchemaProperties` at lines 137/161/248, `forEachSchemaProperties` at line 432), which is what makes this sink-specific rather than a general `ConsumerConfig` limitation.
**Practical impact.** Any client that exposes `schemaProperties` on a sink input spec exposes a write-only field: the value is accepted, dropped, and then reads back absent, producing permanent drift for declarative tooling. This surfaced while reviewing a Terraform provider change (streamnative/terraform-provider-pulsar#224), where exposing the attribute had to be deferred for exactly this reason.
Suggested fix: copy `schemaProperties` in both `convert()` and `convertFromDetails()`, mirroring the `consumerProperties` handling, and add round-trip coverage in `SinkConfigUtilsTest`.
I am happy to put up a PR for this.
### Are you willing to submit a PR?
- [X] I'm willing to submit a PR!
Contributor guide
Research direction
Start in pulsar-functions/utils/src/main/java/org/apache/pulsar/functions/utils/SinkConfigUtils.java, reading convert() and convertFromDetails() alongside the existing consumerProperties handling. Then inspect SinkConfigUtilsTest and add round-trip coverage for the schemaProperties shown in the issue. Done means sink inputSpecs retain schemaProperties in both conversion directions, matching the existing FunctionConfigUtils behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend-api-design
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 88/100