watch等命令 retransformClasses 执行失败 java.lang.InternalError

执行：

```
$ watch watch org.apache.logging.log4j.core.filter.AbstractFilterable isFiltered "{params,returnObj}"
Error during processing the command: null
```

查看`~/logs/arthas/arthas.log`:

```
java.lang.InternalError: null
at sun.instrument.InstrumentationImpl.retransformClasses0(Native Method) ~[na:1.8.0_102]
at sun.instrument.InstrumentationImpl.retransformClasses(InstrumentationImpl.java:144) ~[na:1.8.0_102]
at com.taobao.arthas.core.advisor.Enhancer.enhance(Enhancer.java:304) ~[arthas-core.jar:3.1.1]
at com.taobao.arthas.core.command.monitor200.EnhancerCommand.enhance(EnhancerCommand.java:111) ~[arthas-core.jar:3.1.1]
at com.taobao.arthas.core.command.monitor200.EnhancerCommand.process(EnhancerCommand.java:66) ~[arthas-core.jar:3.1.1]
at com.taobao.arthas.core.shell.command.impl.AnnotatedCommandImpl.process(AnnotatedCommandImpl.java:82) ~[arthas-core.jar:3.1.1]
at com.taobao.arthas.core.shell.command.impl.AnnotatedCommandImpl.access$100(AnnotatedCommandImpl.java:18) ~[arthas-core.jar:3.1.1]
at com.taobao.arthas.core.shell.command.impl.AnnotatedCommandImpl$ProcessHandler.handle(AnnotatedCommandImpl.java:111) ~[arthas-core.jar:3.1.1]
at com.taobao.arthas.core.shell.command.impl.AnnotatedCommandImpl$ProcessHandler.handle(AnnotatedCommandImpl.java:108) ~[arthas-core.jar:3.1.1]
at com.taobao.arthas.core.shell.system.impl.ProcessImpl$CommandProcessTask.run(ProcessImpl.java:370) ~[arthas-core.jar:3.1.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_102]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_102]
at java.lang.Thread.run(Thread.java:766) [na:1.8.0_102]
```

用sc命令来查找这个类，发现有非常多的实现类：

```
$ sc watch org.apache.logging.log4j.core.filter.AbstractFilterable
watch org.apache.logging.log4j.core.appender.AbstractAppender
watch org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender
watch org.apache.logging.log4j.core.appender.AbstractWriterAppender
watch org.apache.logging.log4j.core.appender.AsyncAppender
watch org.apache.logging.log4j.core.appender.ConsoleAppender
watch org.apache.logging.log4j.core.appender.CountingNoOpAppender
watch org.apache.logging.log4j.core.appender.FailoverAppender
watch org.apache.logging.log4j.core.appender.FileAppender
watch org.apache.logging.log4j.core.appender.MemoryMappedFileAppender
watch org.apache.logging.log4j.core.appender.NullAppender
watch org.apache.logging.log4j.core.appender.OutputStreamAppender
watch org.apache.logging.log4j.core.appender.RandomAccessFileAppender
watch org.apache.logging.log4j.core.appender.RollingFileAppender
watch org.apache.logging.log4j.core.appender.RollingRandomAccessFileAppender
watch org.apache.logging.log4j.core.appender.ScriptAppenderSelector
watch org.apache.logging.log4j.core.appender.SmtpAppender
watch org.apache.logging.log4j.core.appender.SocketAppender
watch org.apache.logging.log4j.core.appender.SyslogAppender
watch org.apache.logging.log4j.core.appender.WriterAppender
watch org.apache.logging.log4j.core.appender.db.AbstractDatabaseAppender
watch org.apache.logging.log4j.core.appender.db.jdbc.JdbcAppender
watch org.apache.logging.log4j.core.appender.db.jpa.JpaAppender
watch org.apache.logging.log4j.core.appender.mom.JmsAppender
watch org.apache.logging.log4j.core.appender.mom.jeromq.JeroMqAppender
watch org.apache.logging.log4j.core.appender.mom.kafka.KafkaAppender
watch org.apache.logging.log4j.core.appender.rewrite.RewriteAppender
watch org.apache.logging.log4j.core.appender.routing.RoutingAppender
watch org.apache.logging.log4j.core.async.AsyncLoggerConfig
watch org.apache.logging.log4j.core.async.AsyncLoggerConfig$RootLogger
watch org.apache.logging.log4j.core.config.AbstractConfiguration
watch org.apache.logging.log4j.core.config.AppenderControl
watch org.apache.logging.log4j.core.config.DefaultConfiguration
watch org.apache.logging.log4j.core.config.LoggerConfig
watch org.apache.logging.log4j.core.config.LoggerConfig$RootLogger
watch org.apache.logging.log4j.core.config.NullConfiguration
watch org.apache.logging.log4j.core.config.xml.XmlConfiguration
watch org.apache.logging.log4j.core.filter.AbstractFilterable
```

查看代码，发现执行的是批量增强：

```java
//com.taobao.arthas.core.advisor.Enhancer
// 构建增强器
final Enhancer enhancer = new Enhancer(adviceId, isTracing, skipJDKTrace, enhanceClassSet, methodNameMatcher, affect);
try {
inst.addTransformer(enhancer, true);

// 批量增强
if (GlobalOptions.isBatchReTransform) {
final int size = enhanceClassSet.size();
final Class[] classArray = new Class[size];
arraycopy(enhanceClassSet.toArray(), 0, classArray, 0, size);
if (classArray.length > 0) {
inst.retransformClasses(classArray);
logger.info("Success to batch transform classes: " + Arrays.toString(classArray));
}
} else {
// for each 增强
for (Class clazz : enhanceClassSet) {
try {
inst.retransformClasses(clazz);
logger.info("Success to transform class: " + clazz);
} catch (Throwable t) {
logger.warn("retransform {} failed.", clazz, t);
if (t instanceof UnmodifiableClassException) {
throw (UnmodifiableClassException) t;
} else if (t instanceof RuntimeException) {
throw (RuntimeException) t;
} else {
throw new RuntimeException(t);
}
}
}
}
} finally {
inst.removeTransformer(enhancer);
}
```

试下把批量增强关掉：

```
$ options batch-re-transform false
```

再来执行：

```
$ watch org.apache.logging.log4j.core.filter.AbstractFilterable isFiltered "{params,returnObj}"
Error during processing the command: java.lang.InternalError
```

再查看日志，打印出了具体的类`org.apache.logging.log4j.core.appender.mom.JmsAppender `：

```
01 2019-06-05 19:48:51.500 WARN [as-command-execute-daemon:arthas] [] [] [] retransform class org.apache.logging.log4j.core.appender.mom.JmsAppender failed.
java.lang.InternalError: null
at sun.instrument.InstrumentationImpl.retransformClasses0(Native Method) ~[na:1.8.0_102]
at sun.instrument.InstrumentationImpl.retransformClasses(InstrumentationImpl.java:144) ~[na:1.8.0_102]
at com.taobao.arthas.core.advisor.Enhancer.enhance(Enhancer.java:311) ~[arthas-core.jar:3.1.1]
at com.taobao.arthas.core.command.monitor200.EnhancerCommand.enhance(EnhancerCommand.java:111) [arthas-core.jar:3.1.1]
```

再用jad来查看下源代码：

```
$ jad org.apache.logging.log4j.core.appender.mom.JmsAppender

ClassLoader:
+-org.springframework.boot.loader.LaunchedURLClassLoader@16267862
+-sun.misc.Launcher$AppClassLoader@18b4aac2
+-sun.misc.Launcher$ExtClassLoader@3ecd23d9

Location:
file:/home/admin/xxx/target/exploded/BOOT-INF/lib/log4j-core-2.8.2.jar!/

/*
* Decompiled with CFR 0_132.
*
* Could not load the following classes:
* javax.jms.JMSException
* javax.jms.Message
* javax.jms.MessageProducer
* org.apache.logging.log4j.Logger
* org.apache.logging.log4j.core.Filter
* org.apache.logging.log4j.core.Layout
* org.apache.logging.log4j.core.LogEvent
* org.apache.logging.log4j.core.appender.AbstractAppender
* org.apache.logging.log4j.core.appender.AppenderLoggingException
* org.apache.logging.log4j.core.appender.mom.JmsAppender$1
* org.apache.logging.log4j.core.appender.mom.JmsAppender$Builder
* org.apache.logging.log4j.core.appender.mom.JmsManager
* org.apache.logging.log4j.core.config.plugins.Plugin
* org.apache.logging.log4j.core.config.plugins.PluginAliases
* org.apache.logging.log4j.core.config.plugins.PluginBuilderFactory
*/
package org.apache.logging.log4j.core.appender.mom;

import java.io.Serializable;
import java.util.concurrent.TimeUnit;
import javax.jms.JMSException;
import javax.jms.Message;
import javax.jms.MessageProducer;
import org.apache.logging.log4j.Logger;
import org.apache.logging.log4j.core.Filter;
import org.apache.logging.log4j.core.Layout;
import org.apache.logging.log4j.core.LogEvent;
import org.apache.logging.log4j.core.appender.AbstractAppender;
import org.apache.logging.log4j.core.appender.AppenderLoggingException;
import org.apache.logging.log4j.core.appender.mom.JmsAppender;
import org.apache.logging.log4j.core.appender.mom.JmsManager;
import org.apache.logging.log4j.core.config.plugins.Plugin;
import org.apache.logging.log4j.core.config.plugins.PluginAliases;
import org.apache.logging.log4j.core.config.plugins.PluginBuilderFactory;

@Plugin(name="JMS", category="Core", elementType="appender", printObject=true)
@PluginAliases(value={"JMSQueue", "JMSTopic"})
public class JmsAppender
extends AbstractAppender {
private final JmsManager manager;
private final MessageProducer producer;
private static transient /* synthetic */ boolean[] $jacocoData;

protected JmsAppender(String string, Filter filter, Layout layout, boolean bl, JmsManager jmsManager) throws JMSException {
void layout2;
void filter2;
void ignoreExceptions;
void name;
void manager;
boolean[] arrbl = JmsAppender.$jacocoInit();
super((String)name, (Filter)filter2, (Layout)layout2, (boolean)ignoreExceptions);
this.manager = manager;
arrbl[0] = true;
this.producer = this.manager.createMessageProducer();
arrbl[1] = true;
}

public void append(LogEvent logEvent) {
boolean[] arrbl = JmsAppender.$jacocoInit();
try {
void event;
void message;
Message message2 = this.manager.createMessage(this.getLayout().toSerializable((LogEvent)event));
arrbl[2] = true;
message.setJMSTimestamp(event.getTimeMillis());
arrbl[3] = true;
this.producer.send((Message)message);
}
catch (JMSException message) {
void e;
arrbl[4] = true;
arrbl[5] = true;
throw new AppenderLoggingException((Throwable)e);
}
arrbl[6] = true;
}

public boolean stop(long l, TimeUnit timeUnit) {
void timeout;
void timeUnit2;
boolean[] arrbl = JmsAppender.$jacocoInit();
this.setStopping();
arrbl[7] = true;
boolean bl = super.stop((long)timeout, (TimeUnit)timeUnit2, false);
arrbl[8] = true;
arrbl[9] = true;
this.setStopped();
arrbl[10] = true;
return stopped &= this.manager.stop((long)timeout, (TimeUnit)timeUnit2);
}

@PluginBuilderFactory
public static Builder newBuilder() {
boolean[] arrbl = JmsAppender.$jacocoInit();
arrbl[11] = true;
return new Builder(null);
}

static /* synthetic */ Logger access$100() {
boolean[] arrbl = JmsAppender.$jacocoInit();
arrbl[12] = true;
return LOGGER;
}

private static /* synthetic */ boolean[] $jacocoInit() {
boolean[] arrbl = $jacocoData;
if (arrbl == null) {
Object[] arrobject = new Object[]{-6190699681961106805L, "org/apache/logging/log4j/core/appender/mom/JmsAppender", 13};
UnknownError.$jacocoAccess.equals(arrobject);
arrbl = $jacocoData = (boolean[])arrobject[0];
}
return arrbl;
}
}
```

**发现是被jacoco处理过的**

Start by reproducing the watch command against the affected Log4j classes, then inspect com.taobao.arthas.core.advisor.Enhancer.java around the batch and per-class retransform paths. Compare the failing org.apache.logging.log4j.core.appender.mom.JmsAppender, its jad output, and the JaCoCo instrumentation evidence. Done means the affected watch operation no longer fails with java.lang.InternalError.

Written by the indexing model from the issue text.