apache / apache/pulsar

[Bug] Sink inputSpecs schemaProperties is silently dropped by SinkConfigUtils in both conversion directions

Open Beginner friendly
#26,425 0 comments 0 reactions 0 assignees View on GitHub
area/function type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

**Search before asking**

- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### Version

Reproduced on `master` (`e4d106f6ee`). The same omission is present on the 4.0.x line.

### Minimal reproduce step

Create a sink whose `inputSpecs` entry carries `schemaProperties`:

```json
{
"tenant": "public",
"namespace": "default",
"name": "test-sink",
"inputSpecs": {
"persistent://public/default/in": {
"schemaType": "avro",
"schemaProperties": { "__alwaysAllowNull": "true" },
"consumerProperties": { "application": "billing" }
}
}
}
```

Then read it back:

```
pulsar-admin sinks get --tenant public --namespace default --name test-sink
```

### What did you expect to see?

`schemaProperties` preserved on the round trip, the same way `consumerProperties` is, and the same way it already works for Pulsar Functions.

### What did you see instead?

`schemaProperties` is silently dropped. The API accepts the value, but it never reaches `ConsumerSpec` and is absent from the response. There is no error or warning, so the caller has no signal that the setting was discarded.

### Anything else?

The proto field exists — `Function.proto:111` defines `map schemaProperties = 5` on `ConsumerSpec` — and `ConsumerConfig.schemaProperties` exists on the model. Only the **sink** conversion path fails to use them, in both directions:

- [`SinkConfigUtils.convert()`](https://github.com/apache/pulsar/blob/master/pulsar-functions/utils/src/main/java/org/apache/pulsar/functions/utils/SinkConfigUtils.java#L148-L171) builds each `ConsumerSpec` from the `inputSpecs` entry and copies `schemaType`/`serdeClassName`, `receiverQueueSize`, `cryptoSpec`, `messagePayloadProcessorSpec`, `consumerProperties` and `poolMessages` — but never `schemaProperties`.
- [`SinkConfigUtils.convertFromDetails()`](https://github.com/apache/pulsar/blob/master/pulsar-functions/utils/src/main/java/org/apache/pulsar/functions/utils/SinkConfigUtils.java#L290-L313) restores the same set on the way back, and likewise never restores `schemaProperties`.

`FunctionConfigUtils` handles the field correctly in both directions (`putSchemaProperties` at lines 137/161/248, `forEachSchemaProperties` at line 432), which is what makes this sink-specific rather than a general `ConsumerConfig` limitation.

**Practical impact.** Any client that exposes `schemaProperties` on a sink input spec exposes a write-only field: the value is accepted, dropped, and then reads back absent, producing permanent drift for declarative tooling. This surfaced while reviewing a Terraform provider change (streamnative/terraform-provider-pulsar#224), where exposing the attribute had to be deferred for exactly this reason.

Suggested fix: copy `schemaProperties` in both `convert()` and `convertFromDetails()`, mirroring the `consumerProperties` handling, and add round-trip coverage in `SinkConfigUtilsTest`.

I am happy to put up a PR for this.

### Are you willing to submit a PR?

- [X] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start in pulsar-functions/utils/src/main/java/org/apache/pulsar/functions/utils/SinkConfigUtils.java, reading convert() and convertFromDetails() alongside the existing consumerProperties handling. Then inspect SinkConfigUtilsTest and add round-trip coverage for the schemaProperties shown in the issue. Done means sink inputSpecs retain schemaProperties in both conversion directions, matching the existing FunctionConfigUtils behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend-api-design
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
88/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.