opensearch-project / opensearch-project/data-prepper
Lambda processor should support additional Timeouts
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 374
- Forks
- 354
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
Is your feature request related to a problem? Please describe.
If the lambda function takes longer than 60 seconds, the response events does not get indexed to Opensearch despite the lambda function succeeding.
In the pipeline, I get the error:
] ERROR org.opensearch.dataprepper.plugins.lambda.processor.LambdaProcessor - software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: Read timed out (SDK Attempt Count: 1)
java.util.concurrent.CompletionException: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: Read timed out (SDK Attempt Count: 1)
at software.amazon.awssdk.utils.CompletableFutureUtils.errorAsCompletionException(CompletableFutureUtils.java:64)
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncExecutionFailureExceptionReportingStage.lambda$execute$0(AsyncExecutionFailureExceptionReportingStage.java:51)
at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934)
at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:911)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:78)
at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$executeHttpRequest$3(MakeAsyncHttpRequestStage.java:167)
at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934)
at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:911)
at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: Read timed out (SDK Attempt Count: 1)
at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:130)
at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:95)
at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.retryPolicyDisallowedRetryException(RetryableStageHelper.java:168)
... 25 more
The timeout settings are coming from netty client settings
https://github.com/opensearch-project/data-prepper/blob/main/data-prepper-plugins/aws-lambda/src/main/java/org/opensearch/dataprepper/plugins/lambda/common/client/LambdaClientFactory.java#L35-L37
and there are other configurable Timeout options.
https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/http/nio/netty/NettyNioAsyncHttpClient.Builder.html#readTimeout(java.time.Duration)
Is it possible to make the ReadTimeOut and AcquireTimeout configurable for the NettyNioAsyncHttpClient to unblock our team?
.httpClient(NettyNioAsyncHttpClient.builder()
.maxConcurrency(clientOptions.getMaxConcurrency())
.connectionTimeout(clientOptions.getConnectionTimeout()).build())
.readTimeout(clientOptions.getReadTimeout()) //by adding this option ?
.build())
Currently the default is 60seconds
Describe the solution you'd like
Make readTimeout and acquireTimeout configurable or increase the default to a higher value
Additional context
https://t.corp.amazon.com/ac104702-98c0-4666-99e4-0f9a7658adcc/communication#2ad79f14-84b8-49c7-97d8-f4e53411430c
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in data-prepper-plugins/aws-lambda/src/main/java/org/opensearch/dataprepper/plugins/lambda/common/client/LambdaClientFactory.java, where the Netty client currently configures max concurrency and connection timeout. Read the NettyNioAsyncHttpClient.Builder timeout options and trace how client options are configured. Done means read and acquire timeouts can be configured or their defaults are increased, with the Lambda processor behavior verified for longer-running functions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, java
- Domain
- backend, cloud
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100