[MSHADE-260] shading does not rewrite Lambda deserializers

**[James Moore](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=lumost)** opened **[MSHADE-260](https://issues.apache.org/jira/browse/MSHADE-260?redirect=false)** and commented

In our spark applications, we've encountered issues around shading and lambdas. Where the javac inserted deserialization function ( included below ) will not be updated to reflect the shaded libraries The relocation rule is the following

```xml

com.google.protobuf
com.google.protobuf.v3

```

But while the majority of the signatures are updated to com.google.protobuf.v3 in the class file, the following byte code shows the string where it is not.

```java
(Ljava/lang/Class;Lorg/apache/hadoop/hbase/io/ImmutableBytesWritable;)Lcom/google/protobuf/Message
```

I believe this problem affects any shaded library which has a classed passed into a serialized lambda instance.

```java
private static java.lang.Object $deserializeLambda$(java.lang.invoke.SerializedLambda);
Code:
0: aload_0
1: invokevirtual #64 // Method java/lang/invoke/SerializedLambda.getImplMethodName:()Ljava/lang/String;
4: astore_1
5: iconst_m1
6: istore_2
7: aload_1
8: invokevirtual #70 // Method java/lang/String.hashCode:()I
11: lookupswitch { // 1
1182156382: 28
default: 39
}
28: aload_1
29: ldc #71 // String lambda$convert$60ebd0cc$1
31: invokevirtual #75 // Method java/lang/String.equals:(Ljava/lang/Object;)Z
34: ifeq 39
37: iconst_0
38: istore_2
39: iload_2
40: lookupswitch { // 1
0: 60
default: 143
}
60: aload_0
61: invokevirtual #78 // Method java/lang/invoke/SerializedLambda.getImplMethodKind:()I
64: bipush 6
66: if_icmpne 143
69: aload_0
70: invokevirtual #81 // Method java/lang/invoke/SerializedLambda.getFunctionalInterfaceClass:()Ljava/lang/String;
73: ldc #83 // String org/apache/spark/api/java/function/Function
75: invokevirtual #84 // Method java/lang/Object.equals:(Ljava/lang/Object;)Z
78: ifeq 143
81: aload_0
82: invokevirtual #87 // Method java/lang/invoke/SerializedLambda.getFunctionalInterfaceMethodName:()Ljava/lang/String;
85: ldc #88 // String call
87: invokevirtual #84 // Method java/lang/Object.equals:(Ljava/lang/Object;)Z
90: ifeq 143
93: aload_0
94: invokevirtual #91 // Method java/lang/invoke/SerializedLambda.getFunctionalInterfaceMethodSignature:()Ljava/lang/String;
97: ldc #92 // String (Ljava/lang/Object;)Ljava/lang/Object;
99: invokevirtual #84 // Method java/lang/Object.equals:(Ljava/lang/Object;)Z
102: ifeq 143
105: aload_0
106: invokevirtual #95 // Method java/lang/invoke/SerializedLambda.getImplClass:()Ljava/lang/String;
109: ldc #96 // String StockHubSpotClass
111: invokevirtual #84 // Method java/lang/Object.equals:(Ljava/lang/Object;)Z
114: ifeq 143
117: aload_0
118: invokevirtual #99 // Method java/lang/invoke/SerializedLambda.getImplMethodSignature:()Ljava/lang/String;
121: ldc #101 // String (Ljava/lang/Class;Lorg/apache/hadoop/hbase/io/ImmutableBytesWritable;)Lcom/google/protobuf/Message;
123: invokevirtual #84 // Method java/lang/Object.equals:(Ljava/lang/Object;)Z
126: ifeq 143
129: aload_0
130: iconst_0
131: invokevirtual #105 // Method java/lang/invoke/SerializedLambda.getCapturedArg:(I)Ljava/lang/Object;
134: checkcast #107 // class java/lang/Class
137: invokedynamic #46, 0 // InvokeDynamic #0:call:(Ljava/lang/Class;)Lorg/apache/spark/api/java/function/Function;
142: areturn
143: new #109 // class java/lang/IllegalArgumentException
146: dup
147: ldc #111 // String Invalid lambda deserialization
149: invokespecial #114 // Method java/lang/IllegalArgumentException."":(Ljava/lang/String;)V
152: athrow
```

---

**Affects:** 3.0.0, 3.1.0

**Issue Links:**
- [FLINK-18006](https://issues.apache.org/jira/browse/FLINK-18006) It will throw Invalid lambda deserialization Exception when writing to elastic search with new format
(_**"causes"**_)
- [FLINK-28043](https://issues.apache.org/jira/browse/FLINK-28043) "Invalid lambda deserialization" in AvroParquetReaders
(_**"causes"**_)

3 votes, 5 watchers

No repository files or tests are named. Start by reproducing the Maven Shade Plugin relocation with the serialized-lambda bytecode shown, then trace how relocation handles SerializedLambda method signatures. Done means shaded lambda deserializers reference relocated classes and the affected Spark-style case no longer throws Invalid lambda deserialization.

Written by the indexing model from the issue text.