Azure / Azure/azure-functions-host
[Compute Separation][HP3+HP5A-M1] Add one-worker assignment and readiness APIs
- Dominant language
- C#
- Stars
- 2k
- Forks
- 482
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 36
Description
## Roadmap mapping
| Field | Value |
|---|---|
| Roadmap ID | `HP3` + `HP5A` (M1 subset) |
| Lane | WorkerProxy |
| BYOC milestone | M1 |
| Parent | #11965 |
| Depends on | `HP2` / #11967; align with `HT13` / #11971 before merge |
| Blocks | M1 capacity publication and [internal BYOC #219](https://github.com/serverless-paas-balam/FlexConsumption-BYOC/issues/219); `HP4` proceeds in parallel |
| Pull request | One PR |
## Why this PR exists
WorkerProxy can relay FunctionRpc, but has no worker-pod lifecycle. The platform cannot determine whether the worker
connected, record its app assignment, or observe revisioned state. This PR adds one behavior: a single-worker
assignment/readiness state machine and the three thin M1 HTTP adapters that expose it.
BYOC workers do not start as placeholders and are not specialized by WorkerProxy. The worker starts in its final,
specialized process environment and follows the normal FunctionRpc startup sequence. The runtime remains responsible
for sending the normal `WorkerInitRequest`; WorkerProxy forwards and observes initialization traffic but does not
originate worker initialization or environment reload.
State, normal-startup observation, and adapters stay together because none has an independent M1 consumer. HTTP
forwarding remains `HP4`; host.json, auth, and drain remain out of scope.
## Source material
**Current `dev` baseline**
- `src/Functions.WorkerProxy/WorkerProxyApplication.cs`: slim composition root, options, three listeners, relay, and
existing `/admin/instance/ready` route.
- `WorkerProxyEndpointConfiguration.cs`: management/runtime-gRPC/worker-gRPC listener ownership and port-role checks.
- `WorkerProxyOptions.cs` / validator: AOT-generated configuration.
- `FunctionRpcRelay.cs`, `.Session.cs`, and `FunctionRpcRelayService.cs`: HP2 attach, pump, and terminal-state contracts;
these invariants must not weaken.
- `Functions.WorkerProxy.csproj`: Native AOT, trim, single-file, and configuration-binding analyzers.
- `test/Functions.WorkerProxy.Tests`: existing application, Kestrel, options, and relay fixtures.
**Behavioral reference only**
Use assignment/state behavior from `origin/feature/compute_separation` `WorkerPodStateManager`, `WorkerPodState`,
`WorkerAssignRequest`, `ManagementApiHandlers`, and `WorkerProxyJsonContext`. Do not port proxy-driven specialization,
environment reload, drain/replacement policy, host.json injection, metadata prefetch, auth, logging, or the old
structure.
**Contract**
- `Functions-NoPlaceholders@brettsam/ga_plan/docs/host/platform-control-apis.md`, Worker Pod APIs, updated for BYOC
normal startup.
- `HT13` / #11971.
- BYOC checkpoint [internal BYOC #219](https://github.com/serverless-paas-balam/FlexConsumption-BYOC/issues/219).
## Current state
Already on `dev`: Native-AOT WorkerProxy with deterministic FunctionRpc relay and process-liveness readiness.
Still impossible:
- NNA cannot tell whether the worker stream connected and sent a valid `StartStream`.
- Nothing records worker assignment identity.
- No monotonic worker-pod state exists.
- `HP4` has no finalized capability snapshot from the normal `WorkerInitResponse` path from which to capture or rewrite
`HttpUri`.
## Scope
1. A deterministic single-worker assignment/readiness state machine with strictly monotonic revision.
2. Normal FunctionRpc startup observation over the existing relay; WorkerProxy sends no initialization or reload
messages.
3. One capability-finalizer hook on the normal `WorkerInitResponse`, followed by an immutable capability snapshot.
4. Exactly three new management-listener routes:
- `GET /admin/worker/ready`
- `POST /admin/worker/assign`
- `POST /admin/infra/instanceState`
5. Source-generated, Native-AOT-safe JSON for M1 DTOs.
## Proposed implementation
### 1. Define the M1 state and payloads
Add an internal state model under `src/Functions.WorkerProxy`:
- `WorkerAssignmentState`: `Unassigned`, `Assigned`, `Ready`, `Failed`.
- Published `WorkerPodStatus`: only `None` and `ReadyForRequest`; do not publish drain states before they are designed.
- `WorkerInstanceState` with container type, pod name, monotonic revision, and nested
pod-status/runtime/function-group/always-ready fields.
- `InstanceStatePollRequest` with client revision.
- M1 `WorkerAssignRequest` containing app identity, function group, always-ready flag, environment dictionary, and app
directory. No host.json, key, or secret member.
- A deterministic ordinal equality helper for idempotent assignment.
The environment dictionary and app directory remain in the M1 platform payload, but WorkerProxy does not send them to
the worker. BYOC supplies the final worker process environment and content before worker startup.
Use generic `JsonStringEnumConverter` and omit null optional members.
### 2. Add pod identity configuration
Add only a validated `WorkerProxyOptions.PodName`, supplied by platform configuration with a deterministic test
fallback. Do not add HTTP forwarding options here.
### 3. Add `WorkerPodStateManager`
One singleton owns all mutable assignment/readiness state. Use `System.Threading.Lock` for the state/listener critical
section and an `Interlocked` transition for the one-shot assignment gate.
- Revision starts at zero, increments for every observable transition, and never resets.
- First assignment wins. Equivalent concurrent/repeated assignment receives the same completed outcome; conflicting
assignment returns conflict.
- Worker attach/detach and validated `StartStream` are driven by the relay.
- Worker stream readiness means a live worker stream sent a valid first `StartStream`.
- Published `ReadyForRequest` requires both a live validated worker stream and successful assignment.
- Terminal worker-stream failure after assignment is terminal for M1 and is replayed to equivalent assignment calls.
- `WaitForChangeAsync` returns immediately for stale client revision, otherwise long-polls up to 60 seconds with
run-continuations-asynchronously listeners that are always removed on timeout/cancellation.
### 4. Observe normal FunctionRpc startup without breaking HP2
Extend the relay/session with one relay-owned message-processing step inside the existing inbound reader:
- Accept runtime and worker connections independently as soon as their listeners are ready.
- Require the first worker message to be `StartStream` with a non-empty worker ID.
- Forward `StartStream` immediately, or preserve HP2 ordered buffering until the runtime connects.
- Forward the runtime's normal `WorkerInitRequest` unchanged to the worker.
- Observe the worker's normal `WorkerInitResponse`.
- Invoke `IWorkerCapabilityFinalizer` exactly once on a clone of the successful response before forwarding that response
to the runtime.
- Freeze an immutable finalized capability snapshot for `HP4`.
- Preserve the runtime request ID and all non-capability initialization fields.
- Propagate terminal relay state to readiness and any pending normal-startup observation.
WorkerProxy does not add a proxy-origin send path, request/response correlator, specialization gate, cached replay, or
`FunctionEnvironmentReloadRequest`.
The existing inbound loop remains the sole writer to each destination channel. The processing step returns exactly one
message to forward, allowing `HP4` to rewrite `HttpUri` later without introducing another channel writer.
### 5. Finalize capabilities from normal startup
The normal startup sequence is:
1. BYOC starts the worker in its final specialized environment.
2. Worker connects to WorkerProxy and sends `StartStream`.
3. WorkerProxy forwards `StartStream` to the runtime.
4. Runtime sends its authoritative `WorkerInitRequest`.
5. WorkerProxy forwards the request unchanged.
6. Worker returns `WorkerInitResponse`.
7. WorkerProxy clones the response, invokes `IWorkerCapabilityFinalizer` once, freezes an immutable capability snapshot,
and forwards the finalized response to the runtime.
Register a no-op finalizer here; `HP4` replaces it to capture/rewrite `HttpUri`.
There is no environment-reload capability merge in BYOC normal startup. The worker's normal `WorkerInitResponse`
contains the capability set used by this milestone.
### 6. Add thin handler methods and routes
Keep status mapping unit-testable outside HTTP:
- worker ready: `200` after a live worker stream sends a valid `StartStream`, otherwise `503`;
- assign: `400` invalid body, `409` conflicting assignment, `503` worker not ready, and `200` success/equivalent replay;
- instance state: `200` on changed revision, `204` after no-change timeout.
Map routes only on the management listener using the existing port-role guard and `.AllowAnonymous()` for the approved
M1 beta exception. Leave existing `/admin/instance/ready` unchanged.
### 7. Add source-generated JSON
Add `WorkerProxyJsonContext` entries for every DTO and register it via `ConfigureHttpJsonOptions`. No reflection
fallback, `Dictionary`, or unsourced polymorphism. The AOT publish must remain warning-free.
## Tests and evidence
Add under `test/Functions.WorkerProxy.Tests`:
- state-manager transitions, monotonic revisions, concurrent/idempotent/conflicting assignment, valid
`StartStream`/detach readiness, terminal failure, long-poll timeout/cancel/listener cleanup;
- handler status mapping without an HTTP server;
- route tests proving management-port-only exposure and ready/assign/state behavior;
- normal-startup relay tests proving WorkerProxy originates no init/reload messages, forwards runtime
`WorkerInitRequest`, forwards/finalizes `WorkerInitResponse`, invokes the finalizer exactly once, freezes the
capability snapshot, and propagates terminal session failure;
- tests proving a future finalizer replacement can rewrite capabilities without adding a second relay writer;
- source-generated JSON field/enum/omission tests;
- all existing HP1/HP2 tests unchanged.
Required commands/evidence:
- WorkerProxy project build with warnings as errors;
- linux-x64 Release Native AOT publish with no trim/AOT warnings;
- `Functions.WorkerProxy.Tests` green;
- HT14 standard-product evidence comment.
The downstream exit signal is: worker readiness returns success after valid `StartStream`, assignment returns success,
instance state reports `ReadyForRequest` at a higher revision, and the normal runtime/worker initialization handshake
produces one immutable finalized capability snapshot for `HP4`.
## Explicit exclusions
- Worker specialization and `FunctionEnvironmentReloadRequest`.
- Proxy-originated `WorkerInitRequest`.
- Customer host.json and `host_configuration_json` (`HP3A`).
- Authentication/authorization, JWT/site-token handling, and transport security.
- Drain/stop/replacement states, eviction, reconnect, relink.
- Cached initialization replay and metadata prefetch.
- HTTP listener/destination/probe/rewrite/forwarding (`HP4`); only the finalizer seam lands here.
- Broad logging/metrics/tracing.
- Multi-worker/function groups/pools/request slots.
- Extension gRPC, Durable tunnel, and assembly streaming.
## Standard-product isolation
Changes stay in WorkerProxy and its tests; no Host/Grpc/Server/Client project references WorkerProxy; no standard
service graph, startup, publish, or deps baseline changes; no runtime topology flag. Post evidence to HT14.
## Acceptance criteria
- Worker readiness is `503` until the live worker stream sends a valid `StartStream`, then `200`.
- Assign records the already-specialized worker's platform identity, is idempotent for equal payload, conflicts for
unequal payload, and sends no FunctionRpc message.
- Published instance state is `ReadyForRequest` only when assignment succeeded and the validated worker stream remains
live.
- Runtime-owned normal initialization messages preserve ordering and content.
- Capability finalization and immutable snapshot creation occur exactly once on successful normal
`WorkerInitResponse`.
- Instance state long-polls with strictly monotonic revision.
- All routes are management-listener-only.
- Serialization and publish are AOT-safe and warning-free.
- Existing relay behavior/tests stay unchanged.
## PR stop condition
Stop when one normally started BYOC worker can be probed, assigned, observed through the three routes, and initialized
by the runtime through the transparent relay with one finalized capability snapshot available to `HP4`. Do not add
specialization, environment reload, HTTP forwarding, auth, drain, host.json, Host link routes, or broader management
APIs.
Contributor guide
Assessment
This issue has not been assessed yet.