Support Datadog unix sockets for reporting traces
- Dominant language
- Go
- Stars
- 1.1k
- Forks
- 378
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 74
Description
### **Supporting resources**
- https://buildkite.com/blog/observability-in-the-agent-via-datadog
- https://docs.datadoghq.com/continuous_integration/setup_pipelines/buildkite/
- https://docs.datadoghq.com/developers/dogstatsd/unix_socket/?tab=host
- #1273
### **Is your feature request related to a problem? Please describe.**
Traces aren’t sent to Datadog when a unix socket is used, because Buildkite only support sending traces via HTTP.
So, even though my Buildkite Agent is configured properly to send traces via the volume mounted socket, they will not send, because the Buildkite implementation expects to use HTTP requests, defaulting to `http://localhost:8126/v0.4/traces`
It appears this is a bug!
### **Describe the solution you'd like**
If one of these environment variables are set:
```yaml
- name: DD_TRACE_AGENT_URL
value: "unix:///var/run/datadog/apm.socket"
- name: DD_APM_RECEIVER_SOCKET
value: "unix:///var/run/datadog/apm.socket"
```
Then I expect the Buildkite Agent to use that address to send traces rather than the default `localhost:8126` address.
### **Describe alternatives you've considered**
Hacking together the address by setting "HOST" to `unix` and "PORT" to `///var/run/datadog/apm.socket` (Doesn't work because of a failure to resolve DNS)
### **Additional context**
With these environment variables:
```yaml
- name: BUILDKITE_TRACING_BACKEND
value: "datadog"
- name: DD_ENV
value: "buildkite"
- name: DD_LOGS_INJECTION
value: "true"
- name: DD_APM_ENABLED
value: "true"
- name: DD_TRACE_ENABLED
value: "true"
- name: DD_TRACE_AGENT_URL
value: "unix:///var/run/datadog/apm.socket"
- name: DD_APM_RECEIVER_SOCKET
value: "unix:///var/run/datadog/apm.socket"
- name: DD_PROFILING_ENABLED
value: "true"
- name: DD_VERSION
value: "1.2.2"
```
I receive this error at the end of the Buildkite job:
```bash
2022/04/19 17:45:28 Datadog Tracer v1.28.0 ERROR: lost 1 traces: Post \"http://localhost:8126/v0.4/traces\": dial tcp 127.0.0.1:8126: connect: connection refused (occurred: 19 Apr 22 17:45 UTC)\r
","size":80171,"header_times":[1650390251265893076,1650390251768560569,1650390251771733083,1650390253199885239,1650390253201794984,1650390253622077377,1650390307627264211,1650390318180815416,1650390328697174000]
```
Contributor guide
Research direction
No files or tests are named in the issue. Start by locating the Buildkite Agent's Datadog tracing configuration and the code that posts to http://localhost:8126/v0.4/traces; verify how the DD_TRACE_AGENT_URL and DD_APM_RECEIVER_SOCKET values are handled. Done means traces can be sent through the specified unix socket instead of the default HTTP address.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- networking, observability
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100