apache / apache/pulsar-site

[doc] Function metrics page omits that Python functions can register custom Prometheus collectors

Open
#1,213 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
HTML
Stars
46
Forks
217
Avg merge
1h
Merged PRs (30d)
3

Description

## Which docs page

[`docs/functions-develop-metrics.md`](https://github.com/apache/pulsar-site/blob/main/docs/functions-develop-metrics.md) — "Use metrics to monitor functions".

## What is missing

The page documents `recordMetric` / `record_metric` / `RecordMetric` for the three runtimes and stops there. That leaves out a capability Python functions already have, and it leaves the reader with no way to know that the runtimes differ.

**A Python function can register arbitrary Prometheus collectors today**, with no SDK change, because the instance serves `prometheus_client`'s process-global registry:

```python
# pulsar-functions/instance/src/main/python/prometheus_client_fix.py:50
def start_http_server(port, addr='', registry=core.REGISTRY):

# pulsar-functions/instance/src/main/python/python_instance_main.py:307 — called with the default
prometheus_client_fix.start_http_server(args.metrics_port)
```

`ContextImpl` registers its own summary into that same global by omitting the `registry` argument (`contextimpl.py:67`), and `prometheus_client` defaults every collector to `core.REGISTRY`. So this works and is scraped on the function's existing metrics port:

```python
from prometheus_client import Counter

orders = Counter('my_orders_total', 'Orders processed', ['region'])

def process(input, context):
orders.labels(region='us-east').inc()
return input
```

That matters because `record_metric` only ever writes to a single summary, so counters, gauges, histograms and custom labels are otherwise unreachable. A user reading this page today would conclude none of that is possible.

**Java and Go cannot do this.** Both keep their registry private — Go's is an unexported package variable in `pulsar-function-go/pf`, and Java's `FunctionCollectorRegistry` is internal. Requests to expose them are open at apache/pulsar#26403 (Go) and apache/pulsar#24853 (Java). So this is a genuine per-runtime difference the page should state rather than leave the reader to infer.

## Two caveats worth documenting alongside it

Both are reasons to be explicit rather than reasons to stay silent:

1. **It currently works by convention, not contract.** Nothing declares that the instance serves the global registry, so today it could change without anyone considering it a breaking change. Documenting it is what turns it into a supported behaviour — which is the decision this issue is really asking for.
2. **Metric name collisions fail loudly.** Registering a collector named `pulsar_function_user_metric`, or anything else already registered, raises at registration time. Worth a sentence, and worth noting that the `pulsar_function_` prefix is reserved.

## A smaller inconsistency on the same page

Line 25 says custom metrics are available "for **Java and Python** functions"; line 27 then says "for **Java, Python and Go** functions" and the page includes a Go tab with a working `RecordMetric` example. Go does support `RecordMetric`, so line 25 is the wrong one.

## Suggested resolution

Either:

- **document the Python capability** as supported, with the two caveats above and a note that Java and Go do not have an equivalent (linking apache/pulsar#26403 and apache/pulsar#24853); or
- **state explicitly that it is unsupported** and may change, if maintainers would rather not commit to it.

Either is better than the current silence, which leaves a working capability undiscoverable and simultaneously unprotected from being removed by accident.

Happy to open the PR once there is a steer on which of the two is wanted — that is a maintainer's call rather than a docs detail.

Verified against `apache/pulsar` `origin/master`.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with docs/functions-develop-metrics.md and verify the referenced Python entry points in prometheus_client_fix.py, python_instance_main.py, and contextimpl.py. First get a maintainer decision on documenting the capability or marking it unsupported; done means the page accurately describes the Python and Java/Go difference, caveats, reserved names, linked issues, and the existing Java wording inconsistency.

Written by the indexing model from the issue text.

Assessment

Tech stack
prometheus, python
Domain
documentation
Issue type
Documentation
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.