aws / aws/aws-sdk-java-v2

Calling KinesisAsyncClient#close() throws BlockingOperationException

Aperta
#6,863 1 commento 0 reazioni 0 assegnatari Vedi su GitHub
bug p2
Lingua principale
Java
Stelle
2.6k
Fork
1k
Merge medio
2g 9h
PR unite (30g)
51

Descrizione

### Describe the bug

Hello,

In Hazelcast we use AWS Java SDK in our Kinesis integration. Recently we've migrated from v1 to v2 and our tests became flaky - sometimes calling KinesisAsyncClient.close() results in:
```
[ WARN] [aws-java-sdk-NettyEventLoop-52-10] [i.n.u.c.DefaultPromise]: An exception was thrown by software.amazon.awssdk.http.nio.netty.internal.http2.Http2MultiplexedChannelPool$$Lambda/0x00007f5300911928.operationComplete()
io.netty.util.concurrent.BlockingOperationException: DefaultChannelPromise@4414e1f4(incomplete)
at io.netty.util.concurrent.DefaultPromise.checkDeadLock(DefaultPromise.java:477)
at io.netty.channel.DefaultChannelPromise.checkDeadLock(DefaultChannelPromise.java:159)
at io.netty.util.concurrent.DefaultPromise.awaitUninterruptibly(DefaultPromise.java:283)
at io.netty.channel.DefaultChannelPromise.awaitUninterruptibly(DefaultChannelPromise.java:137)
at io.netty.channel.DefaultChannelPromise.awaitUninterruptibly(DefaultChannelPromise.java:30)
at io.netty.channel.pool.SimpleChannelPool.close(SimpleChannelPool.java:408)
at software.amazon.awssdk.http.nio.netty.internal.BetterSimpleChannelPool.close(BetterSimpleChannelPool.java:38)
at software.amazon.awssdk.http.nio.netty.internal.HonorCloseOnReleaseChannelPool.close(HonorCloseOnReleaseChannelPool.java:80)
at software.amazon.awssdk.http.nio.netty.internal.http2.Http2MultiplexedChannelPool.lambda$doClose$11(Http2MultiplexedChannelPool.java:419)
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:604)
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:571)
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:506)
at io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:199)
at software.amazon.awssdk.http.nio.netty.internal.http2.Http2MultiplexedChannelPool.lambda$doClose$12(Http2MultiplexedChannelPool.java:418)
at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
at io.netty.util.concurrent.PromiseTask.run(PromiseTask.java:106)
at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.base/java.lang.Thread.run(Thread.java:1583)
```

We call close in here: https://github.com/hazelcast/hazelcast/blob/master/extensions/kinesis/src/main/java/com/hazelcast/jet/kinesis/impl/source/KinesisSourcePSupplier.java#L134-L138

In v1 it worked just fine. In v2 is flaky, works sometimes, sometimes throws an exception. Removing a call to `close()` "fixed" the issue, but it really feels rather like a workaround, rather than proper solution - it is said that calling `close()` is not needed, but not that it is prohibited (and if it was, then it would probably throw always).

I see that Flink has similar problem: https://issues.apache.org/jira/browse/FLINK-37949

Our migration commit can be seen here: https://github.com/hazelcast/hazelcast/commit/cb700c3a71214e6a8f84d4cffb33d5e62e1636b4 (if you want to check what has changed in our codebase).

### Regression Issue

- [ ] Select this option if this issue appears to be a regression.

### Expected Behavior

Works always (not only from time to time) and does not throw BlockingOperationException

### Current Behavior

Throws `software.amazon.awssdk.http.nio.netty.internal.http2.Http2MultiplexedChannelPool$$Lambda/0x00007f5300911928.operationComplete()
io.netty.util.concurrent.BlockingOperationException: DefaultChannelPromise`

### Reproduction Steps

You can run [KinesisIntegrationTest#restart_dynamicStream_graceful](https://github.com/hazelcast/hazelcast/blob/master/extensions/kinesis/src/test/java/com/hazelcast/jet/kinesis/KinesisIntegrationTest.java) in Hazelcast codebase in a loop, will crash eventually.

### Possible Solution

_No response_

### Additional Information/Context

_No response_

### AWS Java SDK version used

2.42.31

### JDK version used

Range from JDK17 to JDK25

### Operating System and version

Test running on Linux, not working on Mac 26.3.1 either

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Inizia con Hazelcast's KinesisIntegrationTest#restart_dynamicStream_graceful e la chiamata a close in KinesisSourcePSupplier.java alle righe 134-138 per riprodurre il fallimento in un ciclo. Poi traccia il percorso di close segnalato attraverso Http2MultiplexedChannelPool e BetterSimpleChannelPool. Il lavoro è completato quando KinesisAsyncClient.close() termina in modo consistente senza emettere BlockingOperationException.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
aws, java
Ambito
backend, networking
Tipo di issue
Bug
Difficoltà
4/5
Tempo stimato
3-5 giorni
Stato di attività
Tranquilla
Chiarezza
Abbastanza chiara
Idoneità per principianti
45/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.