aws / aws/aws-sam-cli

Bug: AWS OTEL lambda layer fails with "Cannot register Telemetry API client" when running locally

Open
#4,570 9 comments 4 reactions 0 assignees View on GitHub
maintainer/need-followup stage/bug-repro stage/needs-investigation
Dominant language
Python
Stars
6.7k
Forks
1.2k
Avg merge
1d 10h
Merged PRs (30d)
52

Description

### Description:

I'm using the `arn:aws:lambda:us-west-2:901920570463:layer:aws-otel-python-amd64-ver-1-15-0:1` lambda layer in my SAM app. When I run my SAM app locally I get an error. I don't get this error in Lambda, or with the older `arn:aws:lambda:us-west-2:901920570463:layer:aws-otel-python38-amd64-ver-1-9-1:1` layer.

### Steps to reproduce:

1. Instrument a lambda function with these instructions: https://aws-otel.github.io/docs/getting-started/lambda/lambda-python
2. Use `sam local invoke` to run the function locally.

### Observed result:

```
2023-01-12 12:52:47,785 | Config file location: /Users/hooverd/Projects/example-adot-sam-error/sam/samconfig.toml
2023-01-12 12:52:47,786 | Config file '/Users/hooverd/Projects/example-adot-sam-error/sam/samconfig.toml' does not exist
2023-01-12 12:52:47,801 | Using config file: samconfig.toml, config environment: default
2023-01-12 12:52:47,801 | Expand command line arguments to:
2023-01-12 12:52:47,801 | --template_file=/Users/hooverd/Projects/example-adot-sam-error/sam/template.yaml --event=/Users/hooverd/Projects/example-adot-sam-error/scripts/../events/sample-event.json --parameter_overrides={'HoneycombKey': 'xxx'} --function_logical_id=LambdaFunction --no_event --layer_cache_basedir=/Users/hooverd/.aws-sam/layers-pkg --container_host=localhost --container_host_interface=127.0.0.1
2023-01-12 12:52:47,801 | local invoke command is called
2023-01-12 12:52:47,804 | Collected default values for parameters: {}
2023-01-12 12:52:47,811 | There is no customer defined id or cdk path defined for resource LambdaFunction, so we will use the resource logical id as the resource id
2023-01-12 12:52:47,811 | There is no customer defined id or cdk path defined for resource LambdaFunctionLogs, so we will use the resource logical id as the resource id
2023-01-12 12:52:47,811 | 0 stacks found in the template
2023-01-12 12:52:47,811 | Collected default values for parameters: {}
2023-01-12 12:52:47,817 | There is no customer defined id or cdk path defined for resource LambdaFunction, so we will use the resource logical id as the resource id
2023-01-12 12:52:47,817 | There is no customer defined id or cdk path defined for resource LambdaFunctionLogs, so we will use the resource logical id as the resource id
2023-01-12 12:52:47,817 | 2 resources found in the stack
2023-01-12 12:52:47,817 | Found Serverless function with name='LambdaFunction' and CodeUri='src/'
2023-01-12 12:52:47,817 | --base-dir is not presented, adjusting uri src/ relative to /Users/hooverd/Projects/example-adot-sam-error/sam/template.yaml
2023-01-12 12:52:47,825 | Found one Lambda function with name 'LambdaFunction'
2023-01-12 12:52:47,825 | Invoking app.lambda_handler (python3.8)
2023-01-12 12:52:47,825 | Loading AWS credentials from session with profile 'None'
2023-01-12 12:52:49,838 | Resolving code path. Cwd=/Users/hooverd/Projects/example-adot-sam-error/sam, CodeUri=/Users/hooverd/Projects/example-adot-sam-error/sam/src
2023-01-12 12:52:49,839 | Resolved absolute path to code is /Users/hooverd/Projects/example-adot-sam-error/sam/src
2023-01-12 12:52:49,840 | Code /Users/hooverd/Projects/example-adot-sam-error/sam/src is not a zip/jar file
2023-01-12 12:52:49,841 | arn:aws:lambda:us-west-2:901920570463:layer:aws-otel-python-amd64-ver-1-15-0:1 is already cached. Skipping download
2023-01-12 12:52:49,868 | Skip pulling image and use local one: samcli/lambda:python3.8-x86_64-07fc18c6627356ce5c1bde5f3.

2023-01-12 12:52:49,868 | Mounting /Users/hooverd/Projects/example-adot-sam-error/sam/src as /var/task:ro,delegated inside runtime container
2023-01-12 12:52:50,089 | Starting a timer for 60 seconds for function 'LambdaFunction'
START RequestId: 774cb5c2-88b9-4513-b0d2-57c0c64cf204 Version: $LATEST
{"level":"info","ts":1673549570.505352,"msg":"Launching OpenTelemetry Lambda extension","version":"v0.24.0"}
{"level":"info","ts":1673549570.5659735,"logger":"telemetryAPI.Listener","msg":"Listening for requests","address":":4323"}
{"level":"info","ts":1673549570.5688624,"logger":"telemetryAPI.Client","msg":"Subscribing","baseURL":"http://127.0.0.1:9001/2022-07-01/telemetry"}
{"level":"error","ts":1673549570.5724282,"logger":"telemetryAPI.Client","msg":"Subscription failed"}
{"level":"fatal","ts":1673549570.5728705,"msg":"Cannot register Telemetry API client","error":"request to http://127.0.0.1:9001/2022-07-01/telemetry failed: 404[404 Not Found] 404 page not found\n"}
12 Jan 2023 18:52:50,585 [ERROR] (rapid) Init failed error=exit status 1 InvokeID=
{"level":"info","ts":1673549572.933703,"msg":"Launching OpenTelemetry Lambda extension","version":"v0.24.0"}
{"level":"info","ts":1673549572.9581757,"logger":"telemetryAPI.Listener","msg":"Listening for requests","address":":4323"}
{"level":"info","ts":1673549572.9605553,"logger":"telemetryAPI.Client","msg":"Subscribing","baseURL":"http://127.0.0.1:9001/2022-07-01/telemetry"}
{"level":"error","ts":1673549572.9629943,"logger":"telemetryAPI.Client","msg":"Subscription failed"}
**{"level":"fatal","ts":1673549572.9639678,"msg":"Cannot register Telemetry API client","error":"request to http://127.0.0.1:9001/2022-07-01/telemetry failed: 404[404 Not Found] 404 page not found\n"}**
END RequestId: 0c0a5e6b-0b1b-482a-9346-c77fcad4003e
REPORT RequestId: 0c0a5e6b-0b1b-482a-9346-c77fcad4003e Init Duration: 0.95 ms Duration: 2758.49 ms Billed Duration: 2759 ms Memory Size: 128 MB Max Memory Used: 128 MB
2023-01-12 12:52:53,074 | Cleaning all decompressed code dirs
2023-01-12 12:52:53,075 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2023-01-12 12:52:53,111 | Sending Telemetry: {'metrics': [{'commandRun': {'requestId': 'e9c365d0-817b-4036-9b83-9e991f77e1a8', 'installationId': '372e02f6-1aae-4ee1-90dc-ccea72a6f1a8', 'sessionId': '8a0eac1e-fce8-4155-a9bf-bb85d89707c8', 'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.8.16', 'samcliVersion': '1.68.0', 'awsProfileProvided': False, 'debugFlagProvided': True, 'region': 'us-west-2', 'commandName': 'sam local invoke', 'metricSpecificAttributes': {'projectType': 'CFN', 'gitOrigin': None, 'projectName': '7070f4193a201e26a30113b835bccb914b2976ccf72f2f306aaaef7d1f1033d6', 'initialCommit': None}, 'duration': 5273, 'exitReason': 'success', 'exitCode': 0}}]}
2023-01-12 12:52:53,503 | Telemetry response: 200
2023-01-12 12:52:53,504 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2023-01-12 12:52:53,504 | Sending Telemetry: {'metrics': [{'runtimeMetric': {'requestId': '42c94dda-8dd5-4581-a103-3341f62c9986', 'installationId': '372e02f6-1aae-4ee1-90dc-ccea72a6f1a8', 'sessionId': '8a0eac1e-fce8-4155-a9bf-bb85d89707c8', 'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.8.16', 'samcliVersion': '1.68.0', 'runtimes': ['python3.8']}}]}
2023-01-12 12:52:53,823 | Telemetry response: 200
```

### Expected result:

ADOT lambda layer works.

### Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: MacOS Ventura 13.1 w/ Apple M1 Max
3. `sam --version`: 1.68.0
4. AWS region: us-west-2

`Add --debug flag to command you are running`

I've reproduced the error here: https://github.com/hooverdc/example-adot-sam-error

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.