aws / aws/amazon-cloudwatch-agent-operator

Auto-instrumentation hardcodes cloudwatch-agent.amazon-cloudwatch as a 2-label DNS name

Open Beginner friendly
#392 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
31
Forks
40
Avg merge
6d 15h
Merged PRs (30d)
3

Description

# Auto-instrumentation hardcodes cloudwatch-agent.amazon-cloudwatch as a 2-label DNS name, forcing search-domain expansion and blocking ndots: 1

The default Instrumentation env vars injected by the operator's admission webhook (pkg/instrumentation/defaultinstrumentation.go) point at the CloudWatch agent service using a short, non-fully-qualified hostname:

cloudwatchAgentServiceEndpoint := "cloudwatch-agent.amazon-cloudwatch"

This is used to build OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_TRACES_ENDPOINT, OTEL_AWS_APPLICATION_SIGNALS_EXPORTER_ENDPOINT, OTEL_TRACES_SAMPLER_ARG, OTEL_AWS_JMX_EXPORTER_METRICS_ENDPOINT, and related vars for every Java/Python/.NET/Node.js
pod that gets auto-instrumented. Since the value only has one dot, resolving it depends entirely on the pod's DNS search list (and ndots) to append ..svc.cluster.local / .svc.cluster.local before it succeeds.

This causes several concrete problems:

1. Route53 Resolver query logs are polluted with NXDOMAIN. Every OTLP export from every instrumented pod triggers a chain of failed lookups before the correct search-domain suffix is found, and each failed attempt shows up as an NXDOMAIN entry in
Route53 Resolver query logging. On any cluster with meaningful traffic volume this drowns out real DNS issues and adds noise/cost to log-based DNS monitoring.
2. ndots: 1 cannot be adopted. Lowering ndots from the Kubernetes default of 5 to 1 is a common, recommended tuning to cut down on exactly this kind of wasted search-domain resolution chatter. But because this hostname has only 1 dot, a resolver with
ndots: 1 would try it as an absolute name first, which fails outright since cloudwatch-agent.amazon-cloudwatch isn't a real absolute domain. So this hardcoded value blocks that DNS tuning cluster-wide for anyone using auto-instrumentation.
3. Inconsistent with the Windows path. A few lines below, the Windows headless-service case already uses the fully-qualified form:
cloudwatchAgentServiceEndpoint = "cloudwatch-agent-windows-headless.amazon-cloudwatch.svc.cluster.local"
3. There's no technical reason the Linux default couldn't do the same — the FQDN resolves identically and skips the search-list walk entirely.
4. Pointless looping over the resolver search path. Every single request from every instrumented process re-triggers the same sequential, multi-query DNS walk (per-namespace suffix, then svc.cluster.local, then cluster domain, etc.) to resolve a name
that is fully known in advance. This adds needless latency and load to CoreDNS/Route53 Resolver for a destination the operator itself controls and could just as easily emit as a full FQDN from the start.

Contributor guide

Open the contributing guide

Research direction

Start in pkg/instrumentation/defaultinstrumentation.go and inspect how cloudwatchAgentServiceEndpoint is used to build the injected exporter environment variables. Compare the Linux default with the existing Windows fully qualified endpoint. Done means the Linux auto-instrumentation path emits a fully qualified CloudWatch agent service name and no longer depends on search-domain expansion.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
infrastructure
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.