Calling KinesisAsyncClient#close() throws BlockingOperationException
- Lingua principale
- Java
- Stelle
- 2.6k
- Fork
- 1k
- Merge medio
- 2g 9h
- PR unite (30g)
- 51
Descrizione
### Describe the bug
Hello,
In Hazelcast we use AWS Java SDK in our Kinesis integration. Recently we've migrated from v1 to v2 and our tests became flaky - sometimes calling KinesisAsyncClient.close() results in:
```
[ WARN] [aws-java-sdk-NettyEventLoop-52-10] [i.n.u.c.DefaultPromise]: An exception was thrown by software.amazon.awssdk.http.nio.netty.internal.http2.Http2MultiplexedChannelPool$$Lambda/0x00007f5300911928.operationComplete()
io.netty.util.concurrent.BlockingOperationException: DefaultChannelPromise@4414e1f4(incomplete)
at io.netty.util.concurrent.DefaultPromise.checkDeadLock(DefaultPromise.java:477)
at io.netty.channel.DefaultChannelPromise.checkDeadLock(DefaultChannelPromise.java:159)
at io.netty.util.concurrent.DefaultPromise.awaitUninterruptibly(DefaultPromise.java:283)
at io.netty.channel.DefaultChannelPromise.awaitUninterruptibly(DefaultChannelPromise.java:137)
at io.netty.channel.DefaultChannelPromise.awaitUninterruptibly(DefaultChannelPromise.java:30)
at io.netty.channel.pool.SimpleChannelPool.close(SimpleChannelPool.java:408)
at software.amazon.awssdk.http.nio.netty.internal.BetterSimpleChannelPool.close(BetterSimpleChannelPool.java:38)
at software.amazon.awssdk.http.nio.netty.internal.HonorCloseOnReleaseChannelPool.close(HonorCloseOnReleaseChannelPool.java:80)
at software.amazon.awssdk.http.nio.netty.internal.http2.Http2MultiplexedChannelPool.lambda$doClose$11(Http2MultiplexedChannelPool.java:419)
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:604)
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:571)
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:506)
at io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:199)
at software.amazon.awssdk.http.nio.netty.internal.http2.Http2MultiplexedChannelPool.lambda$doClose$12(Http2MultiplexedChannelPool.java:418)
at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
at io.netty.util.concurrent.PromiseTask.run(PromiseTask.java:106)
at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.base/java.lang.Thread.run(Thread.java:1583)
```
We call close in here: https://github.com/hazelcast/hazelcast/blob/master/extensions/kinesis/src/main/java/com/hazelcast/jet/kinesis/impl/source/KinesisSourcePSupplier.java#L134-L138
In v1 it worked just fine. In v2 is flaky, works sometimes, sometimes throws an exception. Removing a call to `close()` "fixed" the issue, but it really feels rather like a workaround, rather than proper solution - it is said that calling `close()` is not needed, but not that it is prohibited (and if it was, then it would probably throw always).
I see that Flink has similar problem: https://issues.apache.org/jira/browse/FLINK-37949
Our migration commit can be seen here: https://github.com/hazelcast/hazelcast/commit/cb700c3a71214e6a8f84d4cffb33d5e62e1636b4 (if you want to check what has changed in our codebase).
### Regression Issue
- [ ] Select this option if this issue appears to be a regression.
### Expected Behavior
Works always (not only from time to time) and does not throw BlockingOperationException
### Current Behavior
Throws `software.amazon.awssdk.http.nio.netty.internal.http2.Http2MultiplexedChannelPool$$Lambda/0x00007f5300911928.operationComplete()
io.netty.util.concurrent.BlockingOperationException: DefaultChannelPromise`
### Reproduction Steps
You can run [KinesisIntegrationTest#restart_dynamicStream_graceful](https://github.com/hazelcast/hazelcast/blob/master/extensions/kinesis/src/test/java/com/hazelcast/jet/kinesis/KinesisIntegrationTest.java) in Hazelcast codebase in a loop, will crash eventually.
### Possible Solution
_No response_
### Additional Information/Context
_No response_
### AWS Java SDK version used
2.42.31
### JDK version used
Range from JDK17 to JDK25
### Operating System and version
Test running on Linux, not working on Mac 26.3.1 either
Guida per i contributori
Apri la guida per i contributori
Direzione di ricerca
Inizia con Hazelcast's KinesisIntegrationTest#restart_dynamicStream_graceful e la chiamata a close in KinesisSourcePSupplier.java alle righe 134-138 per riprodurre il fallimento in un ciclo. Poi traccia il percorso di close segnalato attraverso Http2MultiplexedChannelPool e BetterSimpleChannelPool. Il lavoro è completato quando KinesisAsyncClient.close() termina in modo consistente senza emettere BlockingOperationException.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- aws, java
- Ambito
- backend, networking
- Tipo di issue
- Bug
- Difficoltà
- 4/5
- Tempo stimato
- 3-5 giorni
- Stato di attività
- Tranquilla
- Chiarezza
- Abbastanza chiara
- Idoneità per principianti
- 45/100