aws / aws/amazon-cloudwatch-agent-operator
Auto-instrumentation hardcodes cloudwatch-agent.amazon-cloudwatch as a 2-label DNS name
- Dominant language
- Go
- Stars
- 31
- Forks
- 40
- Avg merge
- 6d 15h
- Merged PRs (30d)
- 3
Description
# Auto-instrumentation hardcodes cloudwatch-agent.amazon-cloudwatch as a 2-label DNS name, forcing search-domain expansion and blocking ndots: 1
The default Instrumentation env vars injected by the operator's admission webhook (pkg/instrumentation/defaultinstrumentation.go) point at the CloudWatch agent service using a short, non-fully-qualified hostname:
cloudwatchAgentServiceEndpoint := "cloudwatch-agent.amazon-cloudwatch"
This is used to build OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_TRACES_ENDPOINT, OTEL_AWS_APPLICATION_SIGNALS_EXPORTER_ENDPOINT, OTEL_TRACES_SAMPLER_ARG, OTEL_AWS_JMX_EXPORTER_METRICS_ENDPOINT, and related vars for every Java/Python/.NET/Node.js
pod that gets auto-instrumented. Since the value only has one dot, resolving it depends entirely on the pod's DNS search list (and ndots) to append ..svc.cluster.local / .svc.cluster.local before it succeeds.
This causes several concrete problems:
1. Route53 Resolver query logs are polluted with NXDOMAIN. Every OTLP export from every instrumented pod triggers a chain of failed lookups before the correct search-domain suffix is found, and each failed attempt shows up as an NXDOMAIN entry in
Route53 Resolver query logging. On any cluster with meaningful traffic volume this drowns out real DNS issues and adds noise/cost to log-based DNS monitoring.
2. ndots: 1 cannot be adopted. Lowering ndots from the Kubernetes default of 5 to 1 is a common, recommended tuning to cut down on exactly this kind of wasted search-domain resolution chatter. But because this hostname has only 1 dot, a resolver with
ndots: 1 would try it as an absolute name first, which fails outright since cloudwatch-agent.amazon-cloudwatch isn't a real absolute domain. So this hardcoded value blocks that DNS tuning cluster-wide for anyone using auto-instrumentation.
3. Inconsistent with the Windows path. A few lines below, the Windows headless-service case already uses the fully-qualified form:
cloudwatchAgentServiceEndpoint = "cloudwatch-agent-windows-headless.amazon-cloudwatch.svc.cluster.local"
3. There's no technical reason the Linux default couldn't do the same — the FQDN resolves identically and skips the search-list walk entirely.
4. Pointless looping over the resolver search path. Every single request from every instrumented process re-triggers the same sequential, multi-query DNS walk (per-namespace suffix, then svc.cluster.local, then cluster domain, etc.) to resolve a name
that is fully known in advance. This adds needless latency and load to CoreDNS/Route53 Resolver for a destination the operator itself controls and could just as easily emit as a full FQDN from the start.
Contributor guide
Research direction
Start in pkg/instrumentation/defaultinstrumentation.go and inspect how cloudwatchAgentServiceEndpoint is used to build the injected exporter environment variables. Compare the Linux default with the existing Windows fully qualified endpoint. Done means the Linux auto-instrumentation path emits a fully qualified CloudWatch agent service name and no longer depends on search-domain expansion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- infrastructure
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100