Proxy Admin: standardized gRPC management interface on the Proxy
- Dominant language
- Java
- Stars
- 22.6k
- Forks
- 12k
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 27
Description
# [RIP-2] Proxy Admin Standardized Management Interface
## Background
RocketMQ 5.0 moved client access behind the stateless Proxy, but operations still
observe clients through broker-side structures (`ConsumerManager` on the broker,
Remoting-era admin commands). **gRPC clients attached to a Proxy are invisible to
those tools**: the control plane cannot answer *"which SDK clients are online, what
do they subscribe to, are they healthy?"* without indirect metrics heuristics.
RIP-1 (Control Plane 5.0 dashboard, requirement `CLIENT-01`) explicitly depends on
a standard server-side interface to read complete gRPC client data. **There is
currently no such interface on the Proxy.**
## Problem Statement
1. **No dedicated admin surface on the Proxy.** The data-plane `MessagingService`
gRPC service is not designed for control-plane queries (client enumeration,
runtime config, connection control, diagnostics). Operators must SSH into brokers
and run Remoting-era `mqadmin` commands, which cannot see gRPC clients.
2. **No least-privilege authorization for admin operations.** Existing admin
operations share the broker ACL scope; there is no way to grant a read-only
observer vs. a break-glass admin on the Proxy itself.
3. **No cluster-wide client view.** Each Proxy only knows its own clients; there is
no documented mechanism to aggregate client state across all proxies in a
cluster.
4. **No self-service observability.** Admin RPCs are not instrumented with their
own RT / error-rate metrics, so admin-interface health is invisible to
monitoring.
5. **No stable, protocol-pure contract.** Broker-internal wire types leak into
admin tooling, coupling every consumer to internal remoting classes.
## Proposal
Implement **RIP-2: Proxy Admin Standardized Management Interface** — a dedicated,
independent gRPC Admin service on the Proxy, isolated from the data plane, with a
stable backward-compatible proto contract and fine-grained ACL 2.0 authorization.
### Goals
1. A dedicated gRPC Admin service on the Proxy, served on its own port
(`adminGrpcPort`, default **8083**), isolated from the data-plane
`MessagingService`. A global kill switch `proxyAdminEnabled` disables the whole
surface.
2. A stable, backward-compatible proto contract (`ProxyAdminService` +
`AdminService` in `apache/rocketmq/v2/admin.proto`, rocketmq-apis **2.3.0**).
3. First-class authorization under dedicated `proxy.admin.*` ACL 2.0 resources
with read-only (`Get`/`List`) / high-privilege (`Update`/`Delete`/`Pub`)
action separation.
4. The service exposes its own call RT and error-rate metrics (OpenTelemetry).
5. Multi-Proxy semantics: a documented, predictable story for cluster-wide views.
### Non-Goals (this iteration)
- Broker-side quota storage.
- Remoting client kick (Remoting clients remain observable via existing broker
channels; the proto carries a `protocol` field for future coverage).
- A central client registry (proxies are stateless; peer-list configuration is
explicit and auditable).
## Design Decisions
| ID | Decision | Summary |
|----|----------|---------|
| D1 | Service placement | Dedicated `ProxyAdminService` on its own gRPC server/port (8083), separate from data plane and broker-facing `Admin`. Intentionally does NOT expose channelz or proto reflection. |
| D2 | Authorization | Every RPC maps to one `proxy.admin.*` resource + one action. Resources modeled as cluster-typed literals (`cluster:proxy.admin.`). Fail-closed `proxyAdminRequireAuth` mode. Audit logging per served RPC. |
| D3 | Multi-proxy semantics | Each Proxy returns its LOCAL view tagged with `proxy_endpoint` + monotonic `epoch`. `PROXY_SCOPE_ALL_PROXIES` fans out to configured peers in parallel and deduplicates by `client_id`. Peer failures degrade gracefully. |
| D4 | Pagination | Cursor-based `next_token` for client listings (stable under connection churn). Offset pagination (`page_num`/`page_size`, max 100) for bounded diagnostic snapshots. |
| D5 | Protocol coverage | `ClientInstance.protocol` distinguishes GRPC vs REMOTING. This iteration tracks gRPC clients; the field is forward-compatible for Remoting coverage. |
### D2 — Authorization Matrix
| Resource | RPCs | Action |
|----------|------|--------|
| `proxy.admin.client` | ListClients / ListClientsByGroup / ListClientsByTopic | List |
| `proxy.admin.client` | DescribeClient / DescribePopReceiptHandles / DescribeBatchConsumeDiagnostics / ListSubscription / DescribeSubscription / ListConsumerConnection / DescribeGroupAccumulation / GetConsumerRunningInfo / QueryTimeSpan | Get |
| `proxy.admin.config` | DescribeProxyConfig | Get |
| `proxy.admin.config` | UpdateProxyConfig / ChangeLogLevel | Update |
| `proxy.admin.connection` | KickClient / DisconnectChannel / PrintThreadStackTrace / VerifyMessage | Update (high privilege) |
| `proxy.admin.quota` | DescribeQuota | Get |
| `proxy.admin.quota` | UpdateQuota | Update (high privilege) |
| `proxy.admin.route` | DescribeRouteTopology / GetTopicRoute | Get |
| `proxy.admin.route` | SubscribeRouteEvents | List |
| `proxy.admin.ops` | GetProxyRuntimeStats / DescribeTopicStatus / QueryMessage | Get |
| `proxy.admin.ops` | ResetGroupOffset | Update (high privilege) |
| `proxy.admin.ops` | DeleteSubscription | Delete (high privilege) |
| `proxy.admin.ops` | AdminSendMessage | Pub (high privilege) |
### RPC Surface (14+ RPCs across two services)
**`ProxyAdminService` (M1–M4):**
- M1: `ListClients`, `DescribeClient`, `ListClientsByGroup`, `ListClientsByTopic`
- M2: `DescribeProxyConfig` / `UpdateProxyConfig`, `KickClient` / `DisconnectChannel`, `DescribeQuota` / `UpdateQuota`
- M3/M4: `DescribePopReceiptHandles`, `DescribeBatchConsumeDiagnostics`
- Route observation: `SubscribeRouteEvents` (server-streaming), `DescribeRouteTopology`
**`AdminService` (broker-facing, served through the Proxy's managed client):**
- `GetProxyRuntimeStats`, `DescribeTopicStatus`, `QueryMessage`, `QueryTimeSpan`,
`GetConsumerRunningInfo`, `ListConsumerConnection`, `ListSubscription`,
`DescribeSubscription`, `DescribeGroupAccumulation`, `ResetGroupOffset`,
`DeleteSubscription`, `AdminSendMessage`, `PrintThreadStackTrace`,
`VerifyMessage`, `ChangeLogLevel`, `GetTopicRoute`
### Observability
- `rocketmq_proxy_admin_rpc_total{rpc_method, status, error_type?}` — error rate
- `rocketmq_proxy_admin_rpc_latency{rpc_method, status}` (ms histogram) — RT P50/P99
### Configuration Reference
| Key | Default | Meaning |
|-----|---------|---------|
| `proxyAdminEnabled` | true | Kill switch; false = admin server not started |
| `adminGrpcPort` | 8083 | Dedicated admin gRPC port (≤0 disables) |
| `proxyAdminRequireAuth` | false | Fail-closed credential enforcement |
| `proxyAdminPeerEndpoints` | [] | Peer admin endpoints for ALL_PROXIES fan-out |
| `proxyAdminPeerTimeoutMillis` | 3000 | Per-peer fan-out timeout |
| `proxyAdminHeartbeatHistorySize` | 16 | Heartbeat records kept per client |
## Acceptance Criteria
| Criterion | Status |
|-----------|--------|
| RIP document + stable backward-compatible proto contract | `docs/rip-2-proxy-admin.md` + rocketmq-apis `admin.proto` |
| Client query RPCs merged; pagination scales with connection churn | D4 stable cursor; page cost O(pageSize) after sort |
| Independent ACL control, read-only/high-risk separation, least-privilege doc | D2 resources/actions + `docs/rip-2-least-privilege.md` |
| RPC RT & error-rate metrics | ProxyAdminMetricsManager instruments |
| E2E with RIP-1 dashboard | Contract frozen for dashboard CLIENT-01 integration (cross-repo) |
## References
- RIP-1 Control Plane 5.0 dashboard (requirement `CLIENT-01`)
- `docs/rip-2-proxy-admin.md` — full RIP proposal
- `docs/rip-2-least-privilege.md` — least-privilege configuration guide
- rocketmq-apis repository, branch `feature/rip-2-proxy-admin-grpc` (proto contract)
Contributor guide
Research direction
Start with docs/rip-2-proxy-admin.md and docs/rip-2-least-privilege.md, then inspect the rocketmq-apis admin.proto contract on the feature/rip-2-proxy-admin-grpc branch. Map the proposed ProxyAdminService, authorization, pagination, multi-Proxy behavior, and metrics to the acceptance criteria; done requires the listed documents, stable client-query RPCs, ACL separation, metrics, and dashboard contract integration.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- grpc, java
- Domain
- backend-api-design, distributed-systems, documentation, observability, security
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100