[Feat]: Server status collection support for H2O web server.

Problem

Much like many other web servers, the H2O web server provides an option to expose an endpoint which provides runtime performance metrics of the web server itself. Being able to scrape this endpoint for metrics would allow users of H2O to better monitor the performance of the web server.

Description

The de-facto standard (based on the documentation) path for the H2O server status page is /server-status. This actually serves a dynamic HTML page that renders the actual stats, but the stats themselves can be fetched as JSON by instead querying /server-status/json.

Sample JSON data from this endpoint:

{
 
 "server-version": "2.2.6",
 "openssl-version": "OpenSSL 1.1.1q  5 Jul 2022",
 "current-time": "05/Oct/2022:11:25:02 -0400",
 "restart-time": "26/Sep/2022:14:24:57 -0400",
 "uptime": 766805,
 "generation": 1,
 "connections": 1,
 "max-connections": 1024,
 "listeners": 4,
 "worker-threads": 4,
 "num-sessions": 7688,
 "requests": [
  {"host": "::1", "user": null, "at": "20221005T112502.946279-0400", "method": "GET", "path": "/server-status/json", "query": "", "protocol": "HTTP/1.1", "referer": null, "user-agent": "Wget/1.21.3", "connect-time": "0", "request-header-time": "0", "request-body-time": "0", "request-total-time": "0", "process-time": null, "response-time": null, "connection-id": "4453", "ssl.protocol-version": null, "ssl.session-reused": null, "ssl.cipher": null, "ssl.cipher-bits": null, "ssl.session-ticket": null, "http1.request-index": "1", "http2.stream-id": null, "http2.priority.received.exclusive": null, "http2.priority.received.parent": null, "http2.priority.received.weight": null, "http2.priority.actual.parent": null, "http2.priority.actual.weight": null, "authority": "localhost"}
 ],
 "status-errors.400": 3,
 "status-errors.403": 0,
 "status-errors.404": 1140,
 "status-errors.405": 15,
 "status-errors.416": 0,
 "status-errors.417": 0,
 "status-errors.500": 0,
 "status-errors.502": 0,
 "status-errors.503": 0,
 "http2-errors.protocol": 0, 
 "http2-errors.internal": 0, 
 "http2-errors.flow-control": 0, 
 "http2-errors.settings-timeout": 0, 
 "http2-errors.stream-closed": 0, 
 "http2-errors.frame-size": 0, 
 "http2-errors.refused-stream": 0, 
 "http2-errors.cancel": 0, 
 "http2-errors.compression": 0, 
 "http2-errors.connect": 0, 
 "http2-errors.enhance-your-calm": 0, 
 "http2-errors.inadequate-security": 0, 
 "http2.read-closed": 62, 
 "http2.write-closed": 0
,
 "connect-time-0": 0,
 "connect-time-25": 0,
 "connect-time-50": 0,
 "connect-time-75": 0,
 "connect-time-99": 0
, "header-time-0": 0,
 "header-time-25": 0,
 "header-time-50": 0,
 "header-time-75": 0,
 "header-time-99": 0
, "body-time-0": 0,
 "body-time-25": 0,
 "body-time-50": 0,
 "body-time-75": 0,
 "body-time-99": 0
, "request-total-time-0": 0,
 "request-total-time-25": 0,
 "request-total-time-50": 0,
 "request-total-time-75": 0,
 "request-total-time-99": 0
, "process-time-0": 0,
 "process-time-25": 0,
 "process-time-50": 0,
 "process-time-75": 0,
 "process-time-99": 0
, "response-time-0": 0,
 "response-time-25": 0,
 "response-time-50": 0,
 "response-time-75": 0,
 "response-time-99": 0
, "duration-0": 0,
 "duration-25": 0,
 "duration-50": 0,
 "duration-75": 0,
 "duration-99": 0

}

Particularly relevant information form this includes:

• uptime: Tracks time since H2O startup in seconds.
• connections: Gauge of the current number of active connections the server is handling.
• num-sessions: Absolute counter of actual user ‘sessions’ since server startup. AFAICT, this is the number of connections handled, though I’m not certain.
• requests: This is an array with one entry for each request currently in-flight. There’s a lot of per-request info in each entry, but most of it is not super useful as a point-in-time snapshot and overlaps with what we can collect with the web_log collector. A count of currently in-flight requests is, however, possibly very useful.
• status-errors.*: Absolute counters of requests which resulted in specific HTTP error status codes.
• http2-errors.*: Absolute counters of specific types of HTTP/2 errors (these are generally connection related, not request related).
• *-{0,25,50,75,99}: Pre-computed percentiles for specific timing information. I believe these are aggregates just based on the in-flight requests, but I’m not entirely certain (the documentation is unfortunately a bit lacking on descriptions of all of this, and I haven’t had the patience yet to dig into the source code to find out).

Importance

really want

Value proposition

Allow users a similar experience monitoring H2O web servers to what we provide for Apache, Lightspeed, and Nginx.

Proposed implementation

The H2O documentation recommends putting the server status behind some form of authentication, so ideally any implementation needs to be able to handle at least HTTP-Basic authentication.