apache / apache/pulsar

[Python Functions] consumerProperties, poolMessages and messagePayloadProcessorSpec are ignored

Open
#26,410 0 comments 0 reactions 0 assignees View on GitHub
area/function type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

Master Issue: #26412

### Search before reporting

- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

Found while auditing the three function runtimes against `Function.proto` for #26404, which tracks the equivalent gaps in the Go runtime. Filed separately because this is a different runtime and a different fix.

### Motivation

The Python runtime applies `receiverQueueSize`, `schemaProperties`, `serdeClassName`, `schemaType` and `cryptoSpec` from `ConsumerSpec`, but not these three:

```protobuf
map consumerProperties = 6;
bool poolMessages = 8;
MessagePayloadProcessorSpec messagePayloadProcessorSpec = 9;
```

`grep -n "consumerProperties\|poolMessages\|messagePayloadProcessor" python_instance.py contextimpl.py` returns nothing (`Function_pb2.py` excluded, since the generated protobuf contains every field name and makes a naive search report full support).

`consumerProperties` is the one most likely to be noticed: the properties are set on the subscription and read by broker-side tooling, so a function configured with them appears in `pulsar-admin topics stats` without them. `poolMessages` leaves a memory optimisation unavailable. `messagePayloadProcessorSpec` is a Java-centric extension point and may have no reasonable Python equivalent.

The Java runtime applies all three.

### Solution

Pass them through in `setup_consumer`, where the existing per-field handling lives:

- `consumerProperties` → merge into the `properties` dict already passed to `subscribe()`. Worth deciding whether these merge with or override the runtime's own `__pfn_*` properties; the runtime's should win.
- `poolMessages` → confirm whether the Python client exposes an equivalent before promising it.
- `messagePayloadProcessorSpec` → likely out of scope; if so, saying that explicitly is better than continuing to ignore it silently.

### Alternatives

Leaving `messagePayloadProcessorSpec` unimplemented is reasonable. Silently ignoring `consumerProperties` is not, since it is a plain string map with an obvious Python equivalent.

### Anything else?

Verified against `origin/master`.

### Are you willing to submit a PR?

- [X] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start in python_instance.py and contextimpl.py, locating setup_consumer and the existing ConsumerSpec field handling. Compare the Python client API with the Java runtime's treatment of consumerProperties, poolMessages, and messagePayloadProcessorSpec. Done means supported fields are passed through with defined property precedence, while unsupported fields are explicitly documented or scoped out.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.