dotnet / dotnet/aspnetcore

[DirectTls] Add metrics, tracing, and logging on par with the Sockets transport and runtime epoll engine

Open
#67,916 0 comments 0 reactions 1 assignee Claimed by @DeagleGross View on GitHub
area-networking
Dominant language
C#
Stars
38.4k
Forks
10.9k
Avg merge
2d 5h
Merged PRs (30d)
276

Description

Follow-up for the experimental **DirectTls** transport (proposal #67909, draft implementation #67912).

DirectTls currently ships with almost no telemetry. Because it terminates TLS **inside the transport** (on the `TlsEventPump` epoll threads) rather than through `SslStream` in `HttpsConnectionMiddleware`, it both **loses existing observability** and adds a new, unobserved execution surface (the pump). This issue tracks bringing metrics / tracing / logging up to parity with what the **Sockets transport** and the **runtime's epoll socket engine** already provide.

## Motivation / gaps

### 1. TLS handshake metrics are silently lost

`kestrel.tls_handshake.duration` (Histogram) and `kestrel.active_tls_handshakes` (UpDownCounter), on Meter `Microsoft.AspNetCore.Server.Kestrel`, are emitted **only** from `HttpsConnectionMiddleware` (`_metrics.TlsHandshakeStart/Stop(...)`). DirectTls makes that middleware a no-op — the handshake happens on the pump thread against `TlsSocketSession`. So for DirectTls endpoints these two metrics are **never recorded**.

That's the wrong thing to lose for a transport whose entire purpose is TLS performance: handshake latency and in-flight handshake count are exactly what operators need to see. The pump must record the same instruments at handshake begin / complete / fail (including `ConnectionEndReason.TlsHandshakeFailed` on failures), reusing `KestrelMetrics` so the data lands on the existing meter/tags and dashboards keep working.

> Note: transport-agnostic connection metrics (`kestrel.active_connections`, `kestrel.connection.duration`, `kestrel.rejected_connections`, `kestrel.queued_connections`, `kestrel.queued_requests`) are recorded in Kestrel core's connection pipeline and **continue to work** for DirectTls — no action needed there. The gap is specifically the TLS-handshake instruments above.

### 2. Connection lifecycle tracing, including backpressure

The Sockets transport has a full `LoggerMessage`-based trace surface in `SocketsLog.cs` (category `Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets`) that DirectTls has no equivalent for:

| Event (Sockets `SocketsLog`) | Meaning | DirectTls equivalent to add |
| --- | --- | --- |
| `ConnectionPause` / `ConnectionResume` | **backpressure** — input pipe hit `PauseWriterThreshold` / recovered | receive loop pauses/resumes the fd (deregister/re-arm `EPOLLIN`) when `Application.Output` flush backpressures |
| `ConnectionReadFin` | peer half-close (read FIN) | `SSL_read` / `TlsSocketSession` reports EOF |
| `ConnectionWriteFin` / `ConnectionWriteRst` | graceful vs abortive close | send loop / shutdown path |
| `ConnectionError` / `ConnectionReset` | I/O error / RST | pump + `ConnectionIoState` error paths |

**Backpressure is the one we care about most.** In DirectTls the receive loop decrypts on the pump thread and flushes into `Application.Output`; when the app/Kestrel is slow the pipe's `PauseWriterThreshold` trips and the flush stops completing. The pump must then stop reading that fd (drop `EPOLLIN`) and re-arm when the reader drains — and we need to *see* that happen (how often, how long connections sit paused), just like `ConnectionPause`/`ConnectionResume` do for sockets. Without it, a stalled or backpressured pump is invisible.

```mermaid
sequenceDiagram
participant Pump as epoll pump thread
participant Pipe as Application.Output pipe
participant App as Kestrel / app

Pump->>Pipe: FlushAsync decrypted bytes
Note over Pipe: PauseWriterThreshold reached, flush does not complete
Pipe-->>Pump: backpressure - flush pending
Note over Pump: stop reading fd, drop EPOLLIN
log ConnectionPause, bump paused-connections gauge
App->>Pipe: reads and drains
Pipe-->>Pump: ResumeWriterThreshold, flush completes
Note over Pump: re-arm EPOLLIN
log ConnectionResume, clear gauge
```

### 3. Pump / engine health telemetry (inspired by the runtime epoll engine)

The runtime's `SocketAsyncEngine` (Linux epoll) is the direct analogue of our `TlsEventPump` — a small fixed set of dedicated epoll threads pulling readiness events — and `System.Net.Sockets` (`SocketsTelemetry` EventSource) exposes engine-level counters (connections established, bytes sent/received, etc.). We have zero visibility into the pump. We should add engine-level telemetry, e.g.:

- connections currently owned per pump / total, accepts per second
- connections currently paused for backpressure (gauge, ties to §2)
- bytes read / written through the pump
- optional: `epoll_wait` iteration/wakeup counts, ready-event batch sizes, pending-op depth

These can start as an `EventSource` / `EventCounters` surface (like the runtime's), and graduate the operationally-important ones (paused connections, handshake duration) to `System.Diagnostics.Metrics` instruments.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.