aws / aws/aws-sdk-java-v2

Calling KinesisAsyncClient#close() throws BlockingOperationException

Offen
#6,863 1 Kommentar 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
bug p2
Vorherrschende Sprache
Java
Sterne
2.6k
Forks
1k
Ø Merge
2 T. 9 Std.
Gemergte PRs (30 T.)
51

Beschreibung

### Describe the bug

Hello,

In Hazelcast we use AWS Java SDK in our Kinesis integration. Recently we've migrated from v1 to v2 and our tests became flaky - sometimes calling KinesisAsyncClient.close() results in:
```
[ WARN] [aws-java-sdk-NettyEventLoop-52-10] [i.n.u.c.DefaultPromise]: An exception was thrown by software.amazon.awssdk.http.nio.netty.internal.http2.Http2MultiplexedChannelPool$$Lambda/0x00007f5300911928.operationComplete()
io.netty.util.concurrent.BlockingOperationException: DefaultChannelPromise@4414e1f4(incomplete)
at io.netty.util.concurrent.DefaultPromise.checkDeadLock(DefaultPromise.java:477)
at io.netty.channel.DefaultChannelPromise.checkDeadLock(DefaultChannelPromise.java:159)
at io.netty.util.concurrent.DefaultPromise.awaitUninterruptibly(DefaultPromise.java:283)
at io.netty.channel.DefaultChannelPromise.awaitUninterruptibly(DefaultChannelPromise.java:137)
at io.netty.channel.DefaultChannelPromise.awaitUninterruptibly(DefaultChannelPromise.java:30)
at io.netty.channel.pool.SimpleChannelPool.close(SimpleChannelPool.java:408)
at software.amazon.awssdk.http.nio.netty.internal.BetterSimpleChannelPool.close(BetterSimpleChannelPool.java:38)
at software.amazon.awssdk.http.nio.netty.internal.HonorCloseOnReleaseChannelPool.close(HonorCloseOnReleaseChannelPool.java:80)
at software.amazon.awssdk.http.nio.netty.internal.http2.Http2MultiplexedChannelPool.lambda$doClose$11(Http2MultiplexedChannelPool.java:419)
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:604)
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:571)
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:506)
at io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:199)
at software.amazon.awssdk.http.nio.netty.internal.http2.Http2MultiplexedChannelPool.lambda$doClose$12(Http2MultiplexedChannelPool.java:418)
at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
at io.netty.util.concurrent.PromiseTask.run(PromiseTask.java:106)
at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.base/java.lang.Thread.run(Thread.java:1583)
```

We call close in here: https://github.com/hazelcast/hazelcast/blob/master/extensions/kinesis/src/main/java/com/hazelcast/jet/kinesis/impl/source/KinesisSourcePSupplier.java#L134-L138

In v1 it worked just fine. In v2 is flaky, works sometimes, sometimes throws an exception. Removing a call to `close()` "fixed" the issue, but it really feels rather like a workaround, rather than proper solution - it is said that calling `close()` is not needed, but not that it is prohibited (and if it was, then it would probably throw always).

I see that Flink has similar problem: https://issues.apache.org/jira/browse/FLINK-37949

Our migration commit can be seen here: https://github.com/hazelcast/hazelcast/commit/cb700c3a71214e6a8f84d4cffb33d5e62e1636b4 (if you want to check what has changed in our codebase).

### Regression Issue

- [ ] Select this option if this issue appears to be a regression.

### Expected Behavior

Works always (not only from time to time) and does not throw BlockingOperationException

### Current Behavior

Throws `software.amazon.awssdk.http.nio.netty.internal.http2.Http2MultiplexedChannelPool$$Lambda/0x00007f5300911928.operationComplete()
io.netty.util.concurrent.BlockingOperationException: DefaultChannelPromise`

### Reproduction Steps

You can run [KinesisIntegrationTest#restart_dynamicStream_graceful](https://github.com/hazelcast/hazelcast/blob/master/extensions/kinesis/src/test/java/com/hazelcast/jet/kinesis/KinesisIntegrationTest.java) in Hazelcast codebase in a loop, will crash eventually.

### Possible Solution

_No response_

### Additional Information/Context

_No response_

### AWS Java SDK version used

2.42.31

### JDK version used

Range from JDK17 to JDK25

### Operating System and version

Test running on Linux, not working on Mac 26.3.1 either

Beitragsleitfaden

Beitragsleitfaden öffnen

Rechercherichtung

Beginne mit Hazelcast's KinesisIntegrationTest#restart_dynamicStream_graceful und dem close-Aufruf in KinesisSourcePSupplier.java in den Zeilen 134-138, um den Fehler in einer Schleife zu reproduzieren. Verfolge dann den gemeldeten close-Pfad durch Http2MultiplexedChannelPool und BetterSimpleChannelPool. Als abgeschlossen gilt die Aufgabe, wenn KinesisAsyncClient.close() konsistent abgeschlossen wird, ohne BlockingOperationException auszugeben.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
aws, java
Bereich
backend, networking
Issue-Typ
Bug
Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Aktivitätsstatus
Ruhig
Klarheit
Größtenteils klar
Anfängerfreundlichkeit
45/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.