awslabs / awslabs/aws-crt-java
Using AWS CRT client for S3 upload resulting in Thread leak during Integ Test case execution.
- Dominant language
- Java
- Stars
- 76
- Forks
- 46
- PR merge metrics
- No merged PRs in 30d
Description
### Describe the bug
As part of Opensearch s3 plugin. We are migrating from Netty client to crt client to upload files to s3. Our Integration tests have check for ThreadLeaks once the Test Case execution completes. We are seeing ThreadLeak Error on the completion of execution when CRT client is being used.
```
2> com.carrotsearch.randomizedtesting.ThreadLeakError: 3 threads leaked from SUITE scope at org.opensearch.repositories.s3.S3RepositoryThirdPartyTests:
1) Thread[id=83, name=AwsEventLoop 3, state=RUNNABLE, group=main]
at (empty stack)
2) Thread[id=74, name=AwsEventLoop 2, state=RUNNABLE, group=main]
at (empty stack)
3) Thread[id=79, name=AwsEventLoop 8, state=RUNNABLE, group=main]
at (empty stack)
at __randomizedtesting.SeedInfo.seed([13DA5FCB891811FE]:0)
```
on Investigation we are seeing that the native threads used by EventLoopGroup as CRTResource is not getting terminated properly.
Is this known behaviour?
Can something be done to clear these threads?
### Regression Issue
- [ ] Select this option if this issue appears to be a regression.
### Expected Behavior
ThreadLeak should not happen
### Current Behavior
ThreadLeak is seen.
```
2> com.carrotsearch.randomizedtesting.ThreadLeakError: 3 threads leaked from SUITE scope at org.opensearch.repositories.s3.S3RepositoryThirdPartyTests:
1) Thread[id=83, name=AwsEventLoop 3, state=RUNNABLE, group=main]
at (empty stack)
2) Thread[id=74, name=AwsEventLoop 2, state=RUNNABLE, group=main]
at (empty stack)
3) Thread[id=79, name=AwsEventLoop 8, state=RUNNABLE, group=main]
at (empty stack)
at __randomizedtesting.SeedInfo.seed([13DA5FCB891811FE]:0)
```
### Reproduction Steps
Reference Issue for Switching to CRT: https://github.com/opensearch-project/OpenSearch/pull/18800
Reference PR: https://github.com/opensearch-project/OpenSearch/pull/18800
We have currently averted the issue by making "@ThreadLeakScope(ThreadLeakScope.Scope.NONE)"
https://github.com/opensearch-project/OpenSearch/pull/18800/files#diff-0e02bc404b49fa24e689c323473778abf40549bbacd9473ac0c192331e53bf0dR92
### Possible Solution
_No response_
### Additional Information/Context
_No response_
### aws-crt-java version used
2.30.31
### Java version used
21
### Operating System and version
Linux, MAC
Contributor guide
Assessment
This issue has not been assessed yet.