elastic / elastic/apm-agent-nodejs

Lambda layer cannot read agent intake request

Open
#2,994 15 comments 0 reactions 0 assignees View on GitHub
agent-nodejs community triage
Dominant language
JavaScript
Stars
594
Forks
244
Avg merge
1d 8h
Merged PRs (30d)
16

Description

**Describe the bug**

I am experiencing a bug which is affecting our production environment: In a percentage of requests (about 2%), the lambda function that is sending APM data to the lambda layer receives the following error:

```
@message | {"log.level":"error","@timestamp":"2022-10-19T08:28:24.326Z","log.origin":{"file.name":"apmproxy/receiver.go","file.line":119},"message":"Could not read agent intake request body: unexpected EOF","ecs.version":"1.6.0","log.origin.stack_trace":"github.com/elastic/apm-aws-lambda/apmproxy.(*Client).handleIntakeV2Events.func1\n\t/var/lib/jenkins/workspace/ibrary_apm-aws-lambda-mbp_v1.1.0/src/github.com/elastic/apm-aws-lambda/apmproxy/receiver.go:119\nnet/http.HandlerFunc.ServeHTTP\n\t/var/lib/jenkins/workspace/ibrary_apm-aws-lambda-mbp_v1.1.0/.gvm/versions/go1.19.linux.amd64/src/net/http/server.go:2109\nnet/http.(*ServeMux).ServeHTTP\n\t/var/lib/jenkins/workspace/ibrary_apm-aws-lambda-mbp_v1.1.0/.gvm/versions/go1.19.linux.amd64/src/net/http/server.go:2487\nnet/http.serverHandler.ServeHTTP\n\t/var/lib/jenkins/workspace/ibrary_apm-aws-lambda-mbp_v1.1.0/.gvm/versions/go1.19.linux.amd64/src/net/http/server.go:2947\nnet/http.(*conn).serve\n\t/var/lib/jenkins/workspace/ibrary_apm-aws-lambda-mbp_v1.1.0/.gvm/versions/go1.19.linux.amd64/src/net/http/server.go:1991"}
-- | --
@timestamp | 1666168104326
ecs.version | 1.6.0
log.level | error
log.origin.file.line | 119
log.origin.file.name | apmproxy/receiver.go
log.origin.stack_trace | github.com/elastic/apm-aws-lambda/apmproxy.(*Client).handleIntakeV2Events.func1/var/lib/jenkins/workspace/ibrary_apm-aws-lambda-mbp_v1.1.0/src/github.com/elastic/apm-aws-lambda/apmproxy/receiver.go:119net/http.HandlerFunc.ServeHTTP/var/lib/jenkins/workspace/ibrary_apm-aws-lambda-mbp_v1.1.0/.gvm/versions/go1.19.linux.amd64/src/net/http/server.go:2109net/http.(*ServeMux).ServeHTTP/var/lib/jenkins/workspace/ibrary_apm-aws-lambda-mbp_v1.1.0/.gvm/versions/go1.19.linux.amd64/src/net/http/server.go:2487net/http.serverHandler.ServeHTTP/var/lib/jenkins/workspace/ibrary_apm-aws-lambda-mbp_v1.1.0/.gvm/versions/go1.19.linux.amd64/src/net/http/server.go:2947net/http.(*conn).serve/var/lib/jenkins/workspace/ibrary_apm-aws-lambda-mbp_v1.1.0/.gvm/versions/go1.19.linux.amd64/src/net/http/server.go:1991
message | Could not read agent intake request body: unexpected EOF
```

**To Reproduce**

Steps to reproduce the behavior:
This issue is not easily reproductible, as it happens in a small percentage of cases without a clear cause.

**Expected behavior**

The lambda layer receives the APM requests correctly

**Environment (please complete the following information)**

- OS: [e.g. Linux]: AWS Lambda x86_x64
- Node.js version: 16.x
- APM Server version: 1.1.0
- Agent version: 3.39.0

**How are you starting the agent? (please tick one of the boxes)**

- [x] Calling `agent.start()` directly (e.g. `require('elastic-apm-node').start(...)`)
- [ ] Requiring `elastic-apm-node/start` from within the source code
- [ ] Starting node with `-r elastic-apm-node/start`

**Additional context**

This issue has been signaled before [here](https://github.com/elastic/apm-agent-nodejs/issues/2575), however the source of the issue described in that bug is not relevant to my case.

Another interesting fact is that we have a suite of 6 lambda functions, all using the layer, however only one of them encounters this issue.

Normally this should not be a problem, but while the lambda layer hangs, it forces the lambda function to remain active until the lambda timeout kills it, which proves problematic for us in peak-traffic times when the extra open sockets overload our MongoDB connection. I have talked before about this [here](https://github.com/elastic/apm-aws-lambda/issues/130)

Agent config options:

```
apm.start({
environment: process.env.STAGE,
captureBody: 'all',
captureErrorLogStackTraces: 'always',
spanStackTraceMinDuration: '0',
usePathAsTransactionName: true,
serviceName: 'TMv2 - Source',
serviceVersion: pkg.version,
});
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.