elastic / elastic/elastic-serverless-forwarder

Configure boto client to fail fast in case of connectivity issues

Open
#199 1 comment 1 reaction 0 assignees View on GitHub
enhancement Team:Cloud-Monitoring
Dominant language
Python
Stars
38
Forks
50
PR merge metrics
No merged PRs in 30d

Description

**Describe the enhancement:**

By default, boto3 client sets the connection timeout towards AWS services to 60 seconds. It also performs a maximum of 10 attempts, where each retry is delayed with exponential backoff. We should lower both [the `connect_timeout` and the `max_attempt` parameters ](https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html) in order to make the Lambda fail faster in case of non-transient connectivity issues.

**Describe a specific use case for the enhancement or feature:**

With the default values, if the user does not properly configure their ESF's VPC/network environment, it is very likely that the Lambda will run indefinitely up to the maximum lifespan of a Lambda (15 minutes), leading to extra costs for the user and slower issue detection time.

** Related issues **

https://github.com/elastic/elastic-serverless-forwarder/issues/197

** Example log **

```
{"message":"Setting s3 timeout as (60, 60)","ecs":{"version":"1.6.0"},"log":{"logger":"botocore.endpoint","origin":{"file":{"line":408,"name":"endpoint.py"},"function":"create_endpoint"},"original":"Setting s3 timeout as (60, 60)"},"process":{"name":"MainProcess","pid":8,"thread":{"id":139780317878080,"name":"MainThread"}}}

{"message":"Starting new HTTPS connection (1): elastic-serverless-forwarder-davide.s3.eu-west-1.amazonaws.com:443","ecs":{"version":"1.6.0"},"log":{"logger":"urllib3.connectionpool","origin":{"file":{"line":1003,"name":"connectionpool.py"},"function":"_new_conn"},"original":"Starting new HTTPS connection (1): elastic-serverless-forwarder-davide.s3.eu-west-1.amazonaws.com:443"},"process":{"name":"MainProcess","pid":8,"thread":{"id":139780317878080,"name":"MainThread"}}}

retries with exponential backoff....

{"message":"Starting new HTTPS connection (2): elastic-serverless-forwarder-davide.s3.eu-west-1.amazonaws.com:443","ecs":{"version":"1.6.0"},"log":{"logger":"urllib3.connectionpool","origin":{"file":{"line":1003,"name":"connectionpool.py"},"function":"_new_conn"},"original":"Starting new HTTPS connection (1): elastic-serverless-forwarder-davide.s3.eu-west-1.amazonaws.com:443"},"process":{"name":"MainProcess","pid":8,"thread":{"id":139780317878080,"name":"MainThread"}}}
.
.
.
.
.
.
{"message":"Starting new HTTPS connection (10): elastic-serverless-forwarder-davide.s3.eu-west-1.amazonaws.com:443","ecs":{"version":"1.6.0"},"log":{"logger":"urllib3.connectionpool","origin":{"file":{"line":1003,"name":"connectionpool.py"},"function":"_new_conn"},"original":"Starting new HTTPS connection (1): elastic-serverless-forwarder-davide.s3.eu-west-1.amazonaws.com:443"},"process":{"name":"MainProcess","pid":8,"thread":{"id":139780317878080,"name":"MainThread"}}}
--
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.