Agent re-parses the entire CA store on every harvest connection (Linux)

Description

On Linux the agent parses the entire system CA store on every harvest connection rather than once per process. In a production Django + gevent application this accounted for 10.55% of on-CPU time and was the single largest self-time frame in the profile.

Two behaviors combine:

1. The agent deliberately closes its collector connection after each harvest (newrelic/core/application.py, comment: "Force close the socket connection which has been created for this harvest. New connection will be create automatically on the next harvest."). The next harvest therefore builds a new connection and a new SSLContext.

2. HttpClient.__init__ only sets ca_certs when the OS has no resolvable CA file (newrelic/common/agent_http.py). On typical Linux images get_default_verify_paths() does resolve, so ca_bundle_path stays None and ca_certs is never passed to the pool.

With no ca_certs, the vendored urllib3 takes its "load OS default certs" branch on every connect:

# newrelic/packages/urllib3/connection.py
default_ssl_context = False
if self.ssl_context is None:
    default_ssl_context = True
    self.ssl_context = create_urllib3_context(...)
...
if (
    not self.ca_certs
    and not self.ca_cert_dir
    and not self.ca_cert_data
    and default_ssl_context
    and hasattr(context, "load_default_certs")
):
    context.load_default_certs()

So the full CA store is re-read and re-parsed once per harvest connection.

Evidence

py-spy, 300 s at 99 Hz, --nonblocking, against a single gunicorn gevent worker serving real production traffic:

on-CPU samples (gevent hub idle excluded):  1431
SSL/TLS leaf frames:                         189  (13.21%)
  of which load_default_certs / set_default_verify_paths:
                                             151  (10.55%)

caller attribution for those 151 samples:
  100%   connect (newrelic/packages/urllib3/connection.py:417)
  chain: urlopen <- _make_request <- _validate_conn <- connect

All 151 samples are the agent. The workload runs the 5 s flexible harvest plus the 60 s default harvest, across two protocols (collector and OTLP, the latter active because application_logging.forwarding is enabled).

Why ca_bundle_path is not a workaround

Setting ca_bundle_path only moves the work from load_default_certs() to load_verify_locations(). Both parse a CA bundle, so the per-connection cost is unchanged:

load_default_certs()             median 3.351 ms
load_verify_locations(certifi)   median 3.431 ms

Suggested fix

Create the SSLContext once per HttpClient and pass it through the connection kwargs:

connection_kwargs["ssl_context"] = self._shared_ssl_context

HTTPSConnectionPool forwards **conn_kw to HTTPSConnection, and a supplied ssl_context makes default_ssl_context False, skipping the reload. Closing the socket per harvest continues to work; only the certificate parsing gets amortized.

We verified this by injecting a shared context into _connection_kwargs locally: creating five clients produced zero additional load_default_certs() calls, and verification settings were unchanged (verify_mode == CERT_REQUIRED, check_hostname is True, 128 CAs loaded).

Version

• Observed on 10.16.0.
• Still present on main (13.5.0). The block added to agent_http.py since 10.x is Windows-only; the POSIX path is unchanged.

Because ca_bundle_path stays None on Linux, Supportability/Python/Certificate/BundleRequired is never reported, so this path is likely invisible in your own telemetry.

Environment

• Python 3.11, Debian bookworm container
• Django 4.2 + DRF, gunicorn with gevent workers (monkey.patch_all())
• distributed_tracing.enabled = true, application_logging.forwarding.enabled = true