apache / apache/pulsar

[Go Functions] Close the configuration parity gaps with the Java and Python runtimes (master issue)

Open
#26,404 2 comments 0 reactions 0 assignees View on GitHub
area/function type/enhancement
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Search before reporting

- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

This is a master issue. The individual gaps are tracked as the sub-issues listed below; this one exists to hold the comparison that produced them and to say what they have in common.

#26412 is the counterpart for the Python runtime, from the same audit.

### Motivation

The Go function runtime honours a much smaller part of `FunctionDetails` than the Java and Python runtimes do. Most of the difference is silent: the configuration is accepted by `pulsar-admin`, stored by the worker, reported back faithfully by `functions get`, carried into the instance in the protobuf, and then never read.

I audited the three runtimes field by field against `Function.proto`, reading the consuming code rather than grepping for field names — the generated `Function_pb2.py` and `pulsar-function-go/pb` contain every field name, so a naive search reports full support for all three.

| Configuration | Java | Python | Go |
| --- | --- | --- | --- |
| `receiverQueueSize` | yes | yes | yes |
| `subscriptionType` / `subscriptionName` / `subscriptionPosition` | yes | yes | yes |
| `timeoutMs` | yes | yes | yes |
| `compressionType` / `batchBuilder` | yes | yes | yes |
| `userConfig` / `secretsMap` / `logTopic` / `autoAck` | yes | yes | yes |
| `retainOrdering` | yes | yes | yes (#26414) |
| `retainKeyOrdering` | yes | yes | yes (#26414, #26421) |
| **`retryDetails`** (`deadLetterTopic`, `maxMessageRetries`) | yes | **no** (#26397) | **no** |
| `forwardSourceMessageProperty` | yes | yes | **no** |
| `batchingSpec` | yes | yes (#26392) | yes (#26393) |
| `cryptoSpec` (consumer) | yes | yes | **no** |
| `schemaProperties` (consumer) | yes | yes | **no** |
| `poolMessages` / `consumerProperties` / `messagePayloadProcessorSpec` | yes | **no** | **no** |
| `negativeAckRedeliveryDelayMs` | yes | yes (#26413) | yes (#26415) |
| `EFFECTIVELY_ONCE` | yes | yes | panics (explicit) |

Producer `maxPendingMessages` and `maxPendingMessagesAcrossPartitions` are absent from both runtimes on master but are already addressed by the open batching PRs (#26392, #26393), so they are not tracked here. `useThreadLocalProducers` has no equivalent in either client.

**The pattern worth naming.** `EFFECTIVELY_ONCE` is the one the Go runtime gets right:

```go
// pulsar-function-go/pf/instanceConf.go:137
if instanceConf.funcDetails.ProcessingGuarantees == pb.ProcessingGuarantees_EFFECTIVELY_ONCE {
panic("Go instance current not support EFFECTIVELY_ONCE processing guarantees.")
}
```

Unsupported, and it says so. Every other row in that table fails the other way: accepted and ignored. An operator has no signal short of observing the behaviour in production and inferring backwards. Where a gap is not going to be closed soon, refusing it explicitly is strictly better than dropping it silently, and that would be a reasonable interim resolution for several of these.

### Solution

**Progress.** Two of the five below are fixed on master, along with the separately-filed batching gaps; the table above reflects the current state rather than the state at filing.

Close the gaps, tracked individually:

- [x] `retainOrdering` / `retainKeyOrdering` ignored when selecting the subscription type — #26405 *(fixed by #26414 and #26421)*
- [ ] `retryDetails` populated but never applied, so no dead letter policy is created — #26406
- [ ] most of `ConsumerSpec` unapplied: `cryptoSpec`, `schemaProperties`, `consumerProperties`, `poolMessages`, `messagePayloadProcessorSpec` — #26407
- [ ] `forwardSourceMessageProperty` not implemented — #26408
- [x] `negativeAckRedeliveryDelayMs` not applied to the consumer — #26409 *(fixed by #26415)*

Related, filed separately because they are a different runtime and a different fix:

- Python: `poolMessages` / `consumerProperties` / `messagePayloadProcessorSpec` unapplied — #26410
- Python: `negativeAckRedeliveryDelayMs` unapplied — #26411 *(fixed by #26413)*

Filed separately and not duplicated here: #26390 / #26391 (batching — both closed by #26392 / #26393), #26397 (Python dead letter, PR #26400 open), #26403 (Go custom metric collectors, PR #26458 open).

Each sub-issue is independently fixable and none blocks another. Ordering by user impact, `retainKeyOrdering` first: it is the only one in the list whose absence silently breaks a guarantee the user explicitly asked for.

### Alternatives

Fixing these as one change was considered and rejected: they touch different parts of `instance.go`, some need a design decision (whether an unimplementable field should refuse rather than ignore), and a single large PR would be hard to review and hard to revert selectively.

### Anything else?

The audit was done against `origin/master`. Java is treated as the reference implementation throughout — I spot-checked the Java side for the rows that mattered (`PulsarSource` for `retryDetails`, `BatchingUtils` for `batchingSpec`, `JavaInstanceRunnable` for the guards) rather than auditing it exhaustively.

### Are you willing to submit a PR?

- [X] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Treat this as a master issue rather than an implementation task, and start with the linked Go sub-issues, especially #26406, #26407, and #26408. Read the relevant consumers in pulsar-function-go, beginning with pulsar-function-go/pf/instanceConf.go and the runtime code in instance.go; done means each selected gap is resolved in its own focused change and the Go runtime no longer silently ignores the documented configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, java, python
Domain
backend, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.