Azure / Azure/azure-functions-host

Add gRPC communication failure metrics for worker channel diagnostics

Open
#11,595 1 comment 0 reactions 1 assignee Claimed by @kshyju View on GitHub
enhancement Logging and Metrics supportability
Dominant language
C#
Stars
2k
Forks
482
Avg merge
2d 10h
Merged PRs (30d)
36

Description

During investigation of ICM 699060688, we identified a gap in our ability to proactively detect transient gRPC communication failures between the Functions Host and language workers. Currently, when gRPC communication fails (connection refused, timeout, etc.), we lack metrics to aggregate and alert on these failures.

**Background:**
- SLA site ping failure occurred due to transient gRPC communication issue with Java 11 worker
- Issue self-mitigated but we had no visibility into failure frequency
- Similar incidents could occur without detection until they impact customer-facing metrics

**Proposed Changes:**

1. Add new metric constants to `src/WebJobs.Script/Diagnostics/MetricEventNames.cs`:
```csharp
// gRPC communication metrics
public const string WorkerGrpcConnectionFailed = "{0}worker.grpc.connection.failed";
public const string WorkerGrpcMessageSendFailed = "{0}worker.grpc.message.send.failed";
public const string WorkerGrpcInitTimeout = "{0}worker.grpc.init.timeout";
public const string WorkerGrpcTransientError = "{0}worker.grpc.transient.error";
```

2. Emit metrics in `src/WebJobs.Script.Grpc/Channel/GrpcWorkerChannel.cs`:
- `HandleWorkerStartStreamError` → emit `WorkerGrpcConnectionFailed`
- `HandleWorkerInitError` → emit `WorkerGrpcInitTimeout` (when TimeoutException)
- `SendStreamingMessageAsync` failure path → emit `WorkerGrpcMessageSendFailed`

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.