apache / apache/pulsar

[Go Functions] Most of ConsumerSpec is ignored: cryptoSpec, schemaProperties, consumerProperties, poolMessages

Open
#26,407 0 comments 0 reactions 0 assignees View on GitHub
area/function type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

Master Issue: #26404

### Search before reporting

- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### Motivation

`ConsumerSpec` carries nine fields. The Go runtime reads two of them.

```protobuf
message ConsumerSpec {
string schemaType = 1; // read
string serdeClassName = 2; // read (schema selection)
bool isRegexPattern = 3; // read
ReceiverQueueSize receiverQueueSize = 4; // read
map schemaProperties = 5; // NOT read
map consumerProperties = 6; // NOT read
CryptoSpec cryptoSpec = 7; // NOT read
bool poolMessages = 8; // NOT read
MessagePayloadProcessorSpec messagePayloadProcessorSpec = 9; // NOT read
}
```

`git grep -in "CryptoSpec\|SchemaProperties\|ConsumerProperties\|PoolMessages\|MessagePayloadProcessor" -- 'pulsar-function-go/pf/*.go'` returns nothing.

The Python runtime applies `schemaProperties` and `cryptoSpec` (`python_instance.py` builds a crypto key reader through `get_crypto_reader(consumer_conf.cryptoSpec)` and threads `schemaProperties` into `get_schema`), and the Java runtime applies all of them.

`cryptoSpec` is the one with real consequences: a Go function consuming from an encrypted topic gets no crypto key reader, so it cannot decrypt. The rest are quieter — `consumerProperties` are invisible to broker-side tooling that reads them, `poolMessages` leaves a performance option unavailable, and `schemaProperties` are dropped from schema resolution.

### Solution

Apply the remaining `ConsumerSpec` fields in `setupConsumer`, mapping onto `pulsar.ConsumerOptions`. The Go client has an equivalent for most of them:

| `ConsumerSpec` field | `pulsar.ConsumerOptions` |
| --- | --- |
| `schemaProperties` | schema properties on the resolved `Schema` |
| `consumerProperties` | `Properties` |
| `cryptoSpec` | `Decryption` (`*MessageDecryptionInfo`) |
| `poolMessages` | no direct equivalent — needs confirming |
| `messagePayloadProcessorSpec` | no equivalent — likely out of scope |

Grouped into one issue because a single change to `setupConsumer` covers them, but they are separable and a partial fix is useful. Suggested order by impact: `cryptoSpec`, then `consumerProperties` and `schemaProperties`, then the two that may have no Go client equivalent.

Where a field genuinely cannot be supported, refusing it explicitly at startup — the way `EFFECTIVELY_ONCE` is refused at `instanceConf.go:137` — would be better than continuing to ignore it. That is the point made in the master issue.

### Alternatives

Leaving these unimplemented is defensible for `messagePayloadProcessorSpec`, which is a Java-centric extension point. It is not defensible for `cryptoSpec`, where the effect is that an encrypted topic cannot be consumed at all.

### Anything else?

Verified against `origin/master`.

### Are you willing to submit a PR?

- [X] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start in the Go runtime's setupConsumer and inspect how it maps ConsumerSpec onto pulsar.ConsumerOptions. Compare the Python and Java runtime handling of cryptoSpec, schemaProperties, consumerProperties, and poolMessages, and check instanceConf.go:137 for the existing unsupported-field refusal pattern. Done means supported fields are applied or unsupported ones are explicitly rejected, with partial fixes remaining useful.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.