jenkinsci / jenkinsci/remoting
[JENKINS-5703] Exception leaves zombie processes for slaves started by command on master
- Dominant language
- Java
- Stars
- 262
- Forks
- 288
- Avg merge
- 8h 35m
- Merged PRs (30d)
- 4
Description
I have several slaves that are started via the "Launch slave via execution of command on the Master." The command is a bash script that acquires Kerberos credentials and then ssh's over to the slave. Periodically, something happens that kills all the connections launched this way. Regardless, the original bash process is not collected by Hudson, and is left as a zombie or defunct process.
The relevant excerpt from the hudson log appears to be:
```
Feb 17, 2010 12:43:14 PM hudson.remoting.Channel$ReaderThread run
SEVERE: I/O error in channel
java.io.EOFException
at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2554)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
at hudson.remoting.Channel$ReaderThread.run(Channel.java:852)
```
I wonder if the fix is something as simple as adding a proc.destroy() to CommandLauncher.java?:
@Override
public void onClosed(Channel channel, IOException cause) {
if (cause != null) {
cause.printStackTrace(
listener.error(hudson.model.Messages.Slave_Terminated(getTimestamp())));
}
ProcessTree.get().killAll(proc, cookie);
proc.destroy();
}
---
Originally reported by jsiirola, imported from: Exception leaves zombie processes for slaves started by command on master
Raw content of original issue
I have several slaves that are started via the "Launch slave via execution of command on the Master." The command is a bash script that acquires Kerberos credentials and then ssh's over to the slave. Periodically, something happens that kills all the connections launched this way. Regardless, the original bash process is not collected by Hudson, and is left as a zombie or defunct process.
The relevant excerpt from the hudson log appears to be:
Feb 17, 2010 12:43:14 PM hudson.remoting.Channel$ReaderThread run
SEVERE: I/O error in channel <host>
java.io.EOFException
at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2554)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
at hudson.remoting.Channel$ReaderThread.run(Channel.java:852)I wonder if the fix is something as simple as adding a proc.destroy() to CommandLauncher.java?:
@Override
public void onClosed(Channel channel, IOException cause) {
if (cause != null) {
cause.printStackTrace(
listener.error(hudson.model.Messages.Slave_Terminated(getTimestamp())));
}
ProcessTree.get().killAll(proc, cookie);
proc.destroy();
}
1 attachment
- [javapath.png](https://issues.jenkins.io/secure/attachment/30781/javapath.png)
> 
Contributor guide
Research direction
Start with CommandLauncher.java and its onClosed(Channel, IOException) entry point, then compare the EOFException sequence in the remoting log with the process cleanup shown in the issue. Investigate the existing ProcessTree cleanup and proc lifecycle; done means a channel failure no longer leaves the command-launch process as a zombie or defunct process.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100