aws / aws/aws-cli

aws lambda invoke does not complete

Open
#5,783 15 comments 0 reactions 0 assignees View on GitHub
bug lambda p2
Dominant language
Python
Stars
17.3k
Forks
4.6k
Avg merge
1d 2h
Merged PRs (30d)
13

Description

- [x] I've gone though the [User Guide](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html) and the [API reference](https://docs.aws.amazon.com/cli/latest/reference/)
- [x] I've searched for [previous similar issues](https://github.com/aws/aws-cli/issues) and didn't find any solution

**Describe the bug**
`aws lambda invoke` command does not complete even though the lambda run has completed successfully

**SDK version number**
```shell
aws --version
aws-cli/2.1.6 Python/3.7.3 Linux/4.14.200-116.320.amzn1.x86_64 exe/x86_64.debian.9 prompt/off
```

**Platform/OS/Hardware/Device**
Inside debian container running on AWS t3.2xlarge/Amazon Linux

**To Reproduce (observed behavior)**
- Lambda with runtime more than 7 minutes
- `aws --debug lambda invoke --cli-read-timeout 0 --cli-connect-timeout 0 --function-name lambda_name --invocation-type RequestResponse output.log`

**Expected behavior**
cli to complete after lambda execution is done

**Logs/output**
- Command debug log
```
2020-12-04 10:06:08,353 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/2.1.6 Python/3.7.3 Linux/4.14.200-116.320.amzn1.x86_64 exe/x86_64.debian.9
2020-12-04 10:06:08,353 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['--debug', 'lambda', 'invoke', '--cli-read-timeout', '0', '--cli-connect-timeout', '0', '--function-name', '****', '--invocation-type', 'RequestResponse', 'output.log']
2020-12-04 10:06:08,365 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler
2020-12-04 10:06:08,365 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler
2020-12-04 10:06:08,365 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler >
2020-12-04 10:06:08,365 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler
2020-12-04 10:06:08,365 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler
2020-12-04 10:06:08,366 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler
2020-12-04 10:06:08,366 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler
2020-12-04 10:06:08,366 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler >
2020-12-04 10:06:08,366 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler
2020-12-04 10:06:08,366 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.1.6/dist/awscli/data/cli.json
2020-12-04 10:06:08,370 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler
2020-12-04 10:06:08,370 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler
2020-12-04 10:06:08,370 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler
2020-12-04 10:06:08,370 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler
2020-12-04 10:06:08,370 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler
2020-12-04 10:06:08,371 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler
2020-12-04 10:06:08,371 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/2.1.6 Python/3.7.3 Linux/4.14.200-116.320.amzn1.x86_64 exe/x86_64.debian.9 prompt/off
2020-12-04 10:06:08,371 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['--debug', 'lambda', 'invoke', '--cli-read-timeout', '0', '--cli-connect-timeout', '0', '--function-name', '****', '--invocation-type', 'RequestResponse', 'output.log']
2020-12-04 10:06:08,371 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler
2020-12-04 10:06:08,371 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler
2020-12-04 10:06:08,372 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler
2020-12-04 10:06:08,372 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler
2020-12-04 10:06:08,372 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler
2020-12-04 10:06:08,373 - MainThread - botocore.utils - DEBUG - IMDS ENDPOINT: http://169.254.169.254/
2020-12-04 10:06:08,375 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler
2020-12-04 10:06:08,375 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler
2020-12-04 10:06:08,386 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.1.6/dist/botocore/data/lambda/2015-03-31/service-2.json
2020-12-04 10:06:08,401 - MainThread - botocore.hooks - DEBUG - Event building-command-table.lambda: calling handler
2020-12-04 10:06:08,401 - MainThread - botocore.hooks - DEBUG - Event building-command-table.lambda: calling handler
2020-12-04 10:06:08,414 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.1.6/dist/botocore/data/lambda/2015-03-31/waiters-2.json
2020-12-04 10:06:08,415 - MainThread - awscli.clidriver - DEBUG - OrderedDict([('function-name', ), ('invocation-type', ), ('log-type', ), ('client-context', ), ('payload', ), ('qualifier', )])
2020-12-04 10:06:08,415 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.lambda.invoke: calling handler
2020-12-04 10:06:08,416 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.lambda.invoke: calling handler
2020-12-04 10:06:08,416 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.lambda.invoke: calling handler
2020-12-04 10:06:08,416 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.lambda.invoke: calling handler
2020-12-04 10:06:08,428 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.1.6/dist/botocore/data/lambda/2015-03-31/paginators-1.json
2020-12-04 10:06:08,428 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.lambda.invoke: calling handler
2020-12-04 10:06:08,430 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.lambda.invoke.function-name: calling handler
2020-12-04 10:06:08,430 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.lambda.invoke: calling handler
2020-12-04 10:06:08,430 - MainThread - awscli.arguments - DEBUG - Unpacked value of '****' for parameter "function_name": '****'
2020-12-04 10:06:08,430 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.lambda.invoke.invocation-type: calling handler
2020-12-04 10:06:08,430 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.lambda.invoke: calling handler
2020-12-04 10:06:08,430 - MainThread - awscli.arguments - DEBUG - Unpacked value of 'RequestResponse' for parameter "invocation_type": 'RequestResponse'
2020-12-04 10:06:08,430 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.lambda.invoke.log-type: calling handler
2020-12-04 10:06:08,430 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.lambda.invoke.client-context: calling handler
2020-12-04 10:06:08,431 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.lambda.invoke.payload: calling handler
2020-12-04 10:06:08,431 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.lambda.invoke.qualifier: calling handler
2020-12-04 10:06:08,431 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.lambda.invoke.outfile: calling handler
2020-12-04 10:06:08,431 - MainThread - botocore.utils - DEBUG - IMDS ENDPOINT: http://169.254.169.254/
2020-12-04 10:06:08,433 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTP connection (1): 169.254.169.254:80
2020-12-04 10:06:09,436 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTP connection (2): 169.254.169.254:80
2020-12-04 10:06:09,437 - MainThread - urllib3.connectionpool - DEBUG - http://169.254.169.254:80 "GET /latest/meta-data/placement/availability-zone/ HTTP/1.1" 200 13
2020-12-04 10:06:09,439 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: env
2020-12-04 10:06:09,439 - MainThread - botocore.credentials - INFO - Found credentials in environment variables.
2020-12-04 10:06:09,440 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.1.6/dist/botocore/data/endpoints.json
2020-12-04 10:06:09,449 - MainThread - botocore.hooks - DEBUG - Event choose-service-name: calling handler
2020-12-04 10:06:09,453 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.lambda: calling handler
2020-12-04 10:06:09,459 - MainThread - botocore.endpoint - DEBUG - Setting lambda timeout as (None, None)
2020-12-04 10:06:09,461 - MainThread - botocore.hooks - DEBUG - Event provide-client-params.lambda.Invoke: calling handler
2020-12-04 10:06:09,461 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.lambda.Invoke: calling handler
2020-12-04 10:06:09,462 - MainThread - botocore.hooks - DEBUG - Event before-call.lambda.Invoke: calling handler
2020-12-04 10:06:09,462 - MainThread - botocore.endpoint - DEBUG - Making request for OperationModel(name=Invoke) with params: {'url_path': '/2015-03-31/functions/****/invocations', 'query_string': {}, 'method': 'POST', 'headers': {'X-Amz-Invocation-Type': 'RequestResponse', 'User-Agent': 'aws-cli/2.1.6 Python/3.7.3 Linux/4.14.200-116.320.amzn1.x86_64 exe/x86_64.debian.9 prompt/off command/lambda.invoke'}, 'body': b'', 'url': 'https://lambda.eu-central-1.amazonaws.com/2015-03-31/functions/****/invocations', 'context': {'client_region': 'eu-central-1', 'client_config': , 'has_streaming_input': True, 'auth_type': None}}
2020-12-04 10:06:09,463 - MainThread - botocore.hooks - DEBUG - Event request-created.lambda.Invoke: calling handler >
2020-12-04 10:06:09,463 - MainThread - botocore.hooks - DEBUG - Event choose-signer.lambda.Invoke: calling handler
2020-12-04 10:06:09,464 - MainThread - botocore.auth - DEBUG - Calculating signature using v4 auth.
2020-12-04 10:06:09,464 - MainThread - botocore.auth - DEBUG - CanonicalRequest:
POST
/2015-03-31/functions/****/invocations

host:lambda.eu-central-1.amazonaws.com
x-amz-date:20201204T100609Z
x-amz-invocation-type:RequestResponse
x-amz-security-token:****

host;x-amz-date;x-amz-invocation-type;x-amz-security-token
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2020-12-04 10:06:09,464 - MainThread - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20201204T100609Z
20201204/eu-central-1/lambda/aws4_request
c655785ab919c6cba65f96b1a4a7e87915bf0d3798ef04326fbe4de6f0daa43e
2020-12-04 10:06:09,464 - MainThread - botocore.auth - DEBUG - Signature:
6f1f4c51328de9e5bd5fe6dda15c05613f8a98e54373907c1b8256332cd94b3f
2020-12-04 10:06:09,464 - MainThread - botocore.endpoint - DEBUG - Sending http request:
2020-12-04 10:06:09,465 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): lambda.eu-central-1.amazonaws.com:443
```
- Lambda CloudWatch Logs
image

```
REPORT RequestId: 519d7462-6fe5-4bb7-8abe-88648d78c61d Duration: 535808.12 ms Billed Duration: 535809 ms Memory Size: 1024 MB Max Memory Used: 98 MB Init Duration: 133.51 ms
```

**Additional context**
The same command runs perfectly on macOS

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.