jenkinsci / jenkinsci/datadog-plugin

Logs are dropped when Durable Task Use Watching is set to true

Open
#536 0 comments 0 reactions 0 assignees View on GitHub
kind/bug
Dominant language
Java
Stars
37
Forks
58
PR merge metrics
No merged PRs in 30d

Description

**Describe the bug**
With build log collection enabled, when the Jenkins controller runs with `-Dorg.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep.USE_WATCHING=true`, no output from `sh` steps executed on an agent is ever forwarded to Datadog. `echo` steps and `[Pipeline]` framework lines are forwarded normally, and the build's Console Output in Jenkins is complete. Removing the property restores full log forwarding.

**To Reproduce**
1. Run Jenkins 2.568.2 with the controller JVM property `-Dorg.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep.USE_WATCHING=true`, controller with 0 executors, and one permanent SSH build agent.
2. Install Datadog plugin 9.2.0. Configure: Report With `HTTP`, a valid API key, US1 intake URLs, **Enable Log Collection** checked.
3. Create a Pipeline job:
```groovy
pipeline {
agent { label 'agent1' }
stages {
stage('Repro') {
steps {
echo 'DDGAP-ECHO-MARKER'
sh 'echo DDGAP-SH-MARKER-1; sleep 5; echo DDGAP-SH-MARKER-2; sleep 5'
}
}
}
}
```
4. Run the job, wait for log indexing, then compare the build's Console Output with Datadog Log Explorer query `service:jenkins`.
5. Console Output contains all step output:
```
[Pipeline] echo
DDGAP-ECHO-MARKER
[Pipeline] sh
+ echo DDGAP-SH-MARKER-1
DDGAP-SH-MARKER-1
+ sleep 5
+ echo DDGAP-SH-MARKER-2
DDGAP-SH-MARKER-2
+ sleep 5
[Pipeline] }
```
Datadog receives everything except the `sh` step's output — a 10-second hole exactly spanning the step:
```
2026-08-19T14:47:56.584Z [Pipeline] echo
2026-08-19T14:47:56.603Z DDGAP-ECHO-MARKER
2026-08-19T14:47:56.607Z [Pipeline] sh
2026-08-19T14:48:06.910Z [Pipeline] }
```
6. Remove the `USE_WATCHING` property, restart, run the same job again: all 22 console lines including the `sh` output are received by Datadog:
```
2026-08-19T14:50:12.525Z [Pipeline] sh
2026-08-19T14:50:12.961Z + echo DDGAP-SH-MARKER-1
2026-08-19T14:50:12.961Z DDGAP-SH-MARKER-1
2026-08-19T14:50:12.961Z + sleep 5
2026-08-19T14:50:18.303Z + echo DDGAP-SH-MARKER-2
2026-08-19T14:50:18.303Z DDGAP-SH-MARKER-2
2026-08-19T14:50:18.303Z + sleep 5
2026-08-19T14:50:23.626Z [Pipeline] }
```

**Expected behavior**
The complete console log, including `sh` step stdout/stderr, is forwarded to Datadog regardless of the `USE_WATCHING` setting.

**Screenshots**
n/a

**Environment and Versions (please complete the following information):**
- Jenkins 2.568.2 (official `jenkins/jenkins:2.568.2-jdk25` image, Temurin 25)
- Datadog plugin 9.2.0, Report With HTTP (agentless), site US1, log collection enabled, global tags `env:local,version:2.568.2`
- workflow-durable-task-step 1479.v56e587f413a_7; reproduced with durable-task 671.v340ff7959010 and 686.v80ff80875b_82
- Agent: permanent SSH node, Temurin 25 full JDK (`eclipse-temurin:25-jdk-noble` + openssh-server)

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the issue with Jenkins 2.568.2, Datadog plugin 9.2.0, an SSH agent, and USE_WATCHING enabled. Trace the build log collection path for output from Pipeline sh steps and compare it with the complete Jenkins Console Output. Done means sh stdout and stderr are forwarded to Datadog with and without USE_WATCHING.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.