apache / apache/pulsar

[Python Functions] Close the configuration parity gaps with the Java runtime (master issue)

Open
#26,412 1 comment 0 reactions 1 assignee Claimed by @david-streamlio View on GitHub
area/function type/enhancement
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Search before reporting

- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

This is a master issue. The individual gaps are tracked as the sub-issues listed below; this one holds the comparison that produced them. It is the Python counterpart to #26404, which tracks the same audit for the Go runtime.

### Motivation

The Python function runtime is much closer to the Java reference than the Go runtime is, but it still drops a set of `FunctionDetails` fields. As with Go, the failures are silent: the configuration is accepted by `pulsar-admin`, stored by the worker, reported back by `functions get`, carried into the instance in the protobuf, and then never read.

I audited the three runtimes field by field against `Function.proto`, reading the consuming code rather than searching for field names — `Function_pb2.py` is generated and contains every field name, so a naive `grep` reports full support for everything.

| Configuration | Java | **Python** | Go |
| --- | --- | --- | --- |
| `receiverQueueSize` | yes | yes | yes |
| `subscriptionType` / `subscriptionName` / `subscriptionPosition` | yes | yes | yes |
| `timeoutMs` | yes | yes | yes |
| `retainOrdering` / `retainKeyOrdering` | yes | yes | no (#26405) |
| `cryptoSpec` (consumer) | yes | yes | no (#26407) |
| `schemaProperties` (consumer) | yes | yes | no (#26407) |
| `forwardSourceMessageProperty` | yes | yes | no (#26408) |
| `compressionType` / `batchBuilder` | yes | yes | yes |
| `userConfig` / `secretsMap` / `logTopic` / `autoAck` | yes | yes | yes |
| `batchingSpec` | yes | yes (#26392) | yes (#26393) |
| **`retryDetails`** (`deadLetterTopic`, `maxMessageRetries`) | yes | **no** | no (#26406) |
| **`consumerProperties`** | yes | **no** | no |
| **`poolMessages`** | yes | **no** | no |
| **`messagePayloadProcessorSpec`** | yes | **no** | no |
| `negativeAckRedeliveryDelayMs` | yes | yes (#26413) | yes (#26415) |

*(An earlier revision of this issue listed `cleanupSubscription` as a Python gap that Go honoured. Both halves were wrong: it is implemented by the worker in `FunctionActioner`, runtime-agnostically, so no instance runtime implements it and none needs to. The row has been removed.)*

Producer `maxPendingMessages` and `maxPendingMessagesAcrossPartitions` are also unapplied on master, but they are already fixed by the open batching PR and are not tracked separately here.

### Solution

Tracked individually:

- [x] `batchingSpec` ignored, so every function has a fixed 10ms publish-latency floor — #26390 *(fixed by #26392)*
- [ ] `retryDetails` ignored, so no dead letter policy is created — #26397 *(PR open: #26400)*
- [ ] `consumerProperties`, `poolMessages` and `messagePayloadProcessorSpec` ignored — #26410
- [x] `negativeAckRedeliveryDelayMs` ignored, so the 60s client default always applies — #26411 *(fixed by #26413)*

Two of the four are now fixed on master (#26392, #26413). A third, `retryDetails`, has PR #26400 open. The table above reflects the current state rather than the state at filing.

**A note on how these should fail.** The Go runtime refuses `EFFECTIVELY_ONCE` explicitly rather than ignoring it:

```go
// pulsar-function-go/pf/instanceConf.go:137
panic("Go instance current not support EFFECTIVELY_ONCE processing guarantees.")
```

Every row marked "no" above fails the other way: accepted and dropped. Where a field is not going to be supported — `messagePayloadProcessorSpec` is the likely candidate here, being a Java-centric extension point — refusing it, or logging a warning at startup, is better than continuing to accept it silently. That is the same point raised in #26404 and is worth settling once for both runtimes.

### Alternatives

Fixing these as one change was considered and rejected: they touch different parts of `python_instance.py`, two already have separate PRs in review, and `cleanupSubscription` may not belong in the instance at all — it may be the worker's delete path that needs it, which is a different discussion from the consumer-option fields.

### Anything else?

The audit was done against `origin/master`. Java is treated as the reference implementation; I spot-checked the Java side for the rows that mattered (`PulsarSource` for `retryDetails` and the negative-ack delay, `BatchingUtils` for `batchingSpec`, `JavaInstanceRunnable` for the presence guards) rather than auditing it exhaustively.

### Are you willing to submit a PR?

- [X] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.