buildkite / buildkite/agent

Support Datadog unix sockets for reporting traces

Open
#1,615 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
1.1k
Forks
378
Avg merge
2d 6h
Merged PRs (30d)
74

Description

### **Supporting resources**
- https://buildkite.com/blog/observability-in-the-agent-via-datadog
- https://docs.datadoghq.com/continuous_integration/setup_pipelines/buildkite/
- https://docs.datadoghq.com/developers/dogstatsd/unix_socket/?tab=host
- #1273

### **Is your feature request related to a problem? Please describe.**
Traces aren’t sent to Datadog when a unix socket is used, because Buildkite only support sending traces via HTTP.

So, even though my Buildkite Agent is configured properly to send traces via the volume mounted socket, they will not send, because the Buildkite implementation expects to use HTTP requests, defaulting to `http://localhost:8126/v0.4/traces`

It appears this is a bug!

### **Describe the solution you'd like**
If one of these environment variables are set:
```yaml
- name: DD_TRACE_AGENT_URL
value: "unix:///var/run/datadog/apm.socket"
- name: DD_APM_RECEIVER_SOCKET
value: "unix:///var/run/datadog/apm.socket"
```
Then I expect the Buildkite Agent to use that address to send traces rather than the default `localhost:8126` address.

### **Describe alternatives you've considered**
Hacking together the address by setting "HOST" to `unix` and "PORT" to `///var/run/datadog/apm.socket` (Doesn't work because of a failure to resolve DNS)

### **Additional context**
With these environment variables:
```yaml
- name: BUILDKITE_TRACING_BACKEND
value: "datadog"
- name: DD_ENV
value: "buildkite"
- name: DD_LOGS_INJECTION
value: "true"
- name: DD_APM_ENABLED
value: "true"
- name: DD_TRACE_ENABLED
value: "true"
- name: DD_TRACE_AGENT_URL
value: "unix:///var/run/datadog/apm.socket"
- name: DD_APM_RECEIVER_SOCKET
value: "unix:///var/run/datadog/apm.socket"
- name: DD_PROFILING_ENABLED
value: "true"
- name: DD_VERSION
value: "1.2.2"
```
I receive this error at the end of the Buildkite job:
```bash
2022/04/19 17:45:28 Datadog Tracer v1.28.0 ERROR: lost 1 traces: Post \"http://localhost:8126/v0.4/traces\": dial tcp 127.0.0.1:8126: connect: connection refused (occurred: 19 Apr 22 17:45 UTC)\r
","size":80171,"header_times":[1650390251265893076,1650390251768560569,1650390251771733083,1650390253199885239,1650390253201794984,1650390253622077377,1650390307627264211,1650390318180815416,1650390328697174000]
```

Contributor guide

Open the contributing guide

Research direction

No files or tests are named in the issue. Start by locating the Buildkite Agent's Datadog tracing configuration and the code that posts to http://localhost:8126/v0.4/traces; verify how the DD_TRACE_AGENT_URL and DD_APM_RECEIVER_SOCKET values are handled. Done means traces can be sent through the specified unix socket instead of the default HTTP address.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
networking, observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.