jenkinsci / jenkinsci/datadog-plugin
Datadog endpoint for logs should be unbound from metrics/trace host.
- Dominant language
- Java
- Stars
- 37
- Forks
- 58
- PR merge metrics
- No merged PRs in 30d
Description
**Note:**
If you have a feature request, you should [contact support](https://docs.datadoghq.com/help/) so the request can be properly tracked.
**Is your feature request related to a problem? Please describe.**
I've started playing with CI integration in Datadog. Because Datadog accepts neither stats nor process data in their raw form, we force this plugin to send data to an agent for processing. In the current implementation, this demands I setup a custom TCP "logger" (which I do understand to be closer to a proxy)
The agent's TCP connector feature is understandably brittle, given its lack of broad use. It seems to be somewhat of an orphan in configuration. Indeed, after nearly a month's time, support clarified that though `com.datadoghq.ad.logs` may largely be configured as expected when running within Docker (using envvars, labels, annotations)...the TCP listener in practice behaves like a check and requires YAML onboard the container to work properly. Although I'm hoping to simply sidestep the need to build a custom container, bind mount, or the like, I'm curious why the configuration for logging is tightly bound to metric collection. There's obvious benefits for consolidating tagging, but that's all I can come up with.
The TCP submission method does not use encryption and is considered by Datadog to be a backup method, and the proxy does seem even less reliable. It's particularly painful given my experience that when the pipe reaches an error state (presumably when Jenkins is abruptly terminated or another malformed request is sent...or some flavor of un-ACKed RST is sent from one end, the agent appears to do nothing to correct the matter. Exascerbating _that_ is that this plugin writes a `SEVERE` level log after each send error, spamming Jenkins logs 1:1 with any offending job, such that I end up with quite a bit of log bloat on any failure.
I think it's likely that https://github.com/jenkinsci/datadog-plugin/pull/221 didn't appreciate that the appeal of https://github.com/jenkinsci/datadog-plugin/pull/74 extended beyond the lack of a local agent. At their core, however, the gathering of CI metrics and logging are independent activities and I'd expect configuration to reflect that if possible.
**Describe the solution you'd like**
Configuration for CI Metrics remains like present, toggled by a boolean with either unified host/port configuration or preferably (someone's bound to end up with an edge case) separate host/ports for each.
* Logging submission is offered its own host configuration, and is selectable between HTTP and TCP submission
**Describe alternatives you've considered**
* Not collecting logs at all
* Monitoring Datadog and Jenkins alike to keep the TCP flow connection healthy
* Life as a lumberjack.
**Additional context**
During my either quixotic or dramatically ill-informed attempt to get the TCP listener to be generated promptly, I've run across a fairly wide swath of troubled souls who had some flavor of trouble setting this up as such. Plenty of unreliable TCP connections, a few issues getting the custom log uptake to start properly, and [the occasional local networking frustration](https://github.com/DataDog/datadog-agent/issues/5958). This approach would retain CI functionality while allowing a user to sidestep what I believe is a needless binding of the two configurations.
Contributor guide
Research direction
Review pull requests #221 and #74 first to understand the current relationship between CI metrics and logging configuration. Then trace the existing configuration entry points for metrics and log submission. Done means logging has independent host configuration with selectable HTTP or TCP submission while CI metrics retain their current configuration.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- observability-sre
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100