alibaba / alibaba/arthas

调用http API retransform接口,概率性出现Cannot interrupt process in TERMINATED state

Open
#2,980 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
37.6k
Forks
7.6k
Avg merge
1d 22h
Merged PRs (30d)
4

Description

- [ ] 我已经在 [issues](https://github.com/alibaba/arthas/issues) 里搜索,没有重复的issue。

### 环境信息

* `arthas-boot.jar` 或者 `as.sh` 的版本:3.7.1
* Arthas 版本: 3.7.1
* 操作系统版本: bclinux
* 目标进程的JVM版本: jdk1.8
* 执行`arthas-boot`的版本: 3.7.1

### 重现问题的步骤

1. 调用http API retransform接口,概率性出现Cannot interrupt process in TERMINATED state
2. 概率性出现,不是每次都出现

### 期望的结果

retransform热部署成功

### 实际运行的结果

retransform热部署未成功,出现查询class是成功的,实际运行代码没有成功。

```
2025-01-04 09:34:51 [arthas-NettyWebsocketTtyBootstrap-4-1] INFO c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0x7b5588b9, L:/7.227.10.168:8563] READ: [id: 0x8f5767d2, L:/7.227.10.168:8563 - R:/10.165.36.20:63971]
2025-01-04 09:34:51 [arthas-NettyWebsocketTtyBootstrap-4-1] INFO c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0x7b5588b9, L:/7.227.10.168:8563] READ COMPLETE
2025-01-04 09:34:51 [arthas-command-execute] INFO c.t.a.c.c.k.RetransformCommand -Try retransform class name: org.cache.OutClient, ClassLoader: ExtensionWebappClassLoader
context: bes
delegate: false
----------> Parent Classloader:
java.net.URLClassLoader@17a7f733

2025-01-04 09:34:51 [arthas-command-execute] INFO c.t.a.c.c.k.RetransformCommand -RetransformCommand match class: org.cache.OutClient, id: 12, classLoaderClass: null, hashCode: null
2025-01-04 09:34:52 [arthas-TermServer-1-17] ERROR c.t.a.c.s.t.i.h.api.HttpApiHandler -process http api request failed: Cannot interrupt process in TERMINATED state
java.lang.IllegalStateException: Cannot interrupt process in TERMINATED state
at com.taobao.arthas.core.shell.system.impl.ProcessImpl.interrupt(ProcessImpl.java:156)
at com.taobao.arthas.core.shell.system.impl.ProcessImpl.interrupt(ProcessImpl.java:138)
at com.taobao.arthas.core.shell.system.impl.JobImpl.interrupt(JobImpl.java:58)
at com.taobao.arthas.core.shell.session.impl.SessionManagerImpl.removeSession(SessionManagerImpl.java:82)
at com.taobao.arthas.core.shell.term.impl.http.api.HttpApiHandler.processExecRequest(HttpApiHandler.java:470)
at com.taobao.arthas.core.shell.term.impl.http.api.HttpApiHandler.dispatchRequest(HttpApiHandler.java:273)
at com.taobao.arthas.core.shell.term.impl.http.api.HttpApiHandler.processRequest(HttpApiHandler.java:254)
at com.taobao.arthas.core.shell.term.impl.http.api.HttpApiHandler.handle(HttpApiHandler.java:103)
at com.taobao.arthas.core.shell.term.impl.http.HttpRequestHandler.channelRead0(HttpRequestHandler.java:72)
at com.taobao.arthas.core.shell.term.impl.http.HttpRequestHandler.channelRead0(HttpRequestHandler.java:31)
at com.alibaba.arthas.deps.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at com.alibaba.arthas.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at com.alibaba.arthas.deps.io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:61)
at com.alibaba.arthas.deps.io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:425)
at com.alibaba.arthas.deps.io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)
at com.alibaba.arthas.deps.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)
at com.alibaba.arthas.deps.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
at com.alibaba.arthas.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
at com.alibaba.arthas.deps.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at com.alibaba.arthas.deps.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at com.alibaba.arthas.deps.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:750)
```

Contributor guide

Open the contributing guide

Research direction

Start at HttpApiHandler.processExecRequest and follow session cleanup into SessionManagerImpl.removeSession and ProcessImpl.interrupt. Reproduce the HTTP retransform request on Arthas 3.7.1 with JDK 8, using the provided stack trace to inspect the terminated-process path. Done means retransform succeeds without the TERMINATED-state error and the new code is actually running.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
devtools
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.