elastic / elastic/ecs-logging-nodejs

ecs-pino-format does not integrate with elastic-apm-node when it is wrapped inside a library

Open
#130 1 comment 2 reactions 0 assignees View on GitHub
agent-nodejs community triage
Dominant language
JavaScript
Stars
69
Forks
45
PR merge metrics
No merged PRs in 30d

Description

Hi. We have an internal NPM package for logging, which wraps Pino and ecs-pino-format to format logs. We are using this library in multiple services. We also use elastic-apm-node in these services.

The problem is that the automatic integration of elastic-apm-node doesn't work when ecs-pino-format is wrapped inside this library, but it works when I directly import Pino and log from the service. `trace.id` and `event.dataset` fields are not added to the logs that come from the library.

What I'm using:

Typescript in both the service and the library.

"@elastic/ecs-pino-format": "^1.1.2"
"elastic-apm-node": "^3.9.0"

Example to explain what I mean:

I created 2 folders, one to simulate the service and the other to simulate the library. I `npm link`ed the library with the service.

pino-format-issue-service:

![image](https://user-images.githubusercontent.com/15969085/197735724-f3211baf-2780-4bea-b61d-40b1934e11ed.png)

pino-format-issue-lib:

![image](https://user-images.githubusercontent.com/15969085/197735917-3b901a42-205b-47ee-9e5e-1e1d71a85ca0.png)

Log output:

```
{"log.level":"info","@timestamp":"2022-10-25T09:16:28.411Z","process":{"pid":71004},"host":{"hostname":""},"ecs":{"version":"1.6.0"},"service":{"name":"test-service"},"event":{"dataset":"test-service.log"},"message":"info from pino"}
{"log.level":"info","@timestamp":"2022-10-25T09:16:28.412Z","process":{"pid":71004},"host":{"hostname":""},"ecs":{"version":"1.6.0"},"message":"info from lib"}
```
As you can see, `event.dataset` field is added to the first log (directly logging from `pino`), but it's not there in the second log.

What I've tried:

1. Importing `elastic-apm-node/start` in both the service and the library.
This adds `trace.id` to the logs from library as well, but the ids from the service and library do not match as the agent is imported twice. This defeats the purpose as I will not be able to jump from APM to the log.

2. Importing the agent only inside the library, and removing it from the service.
I would be fine with doing this, but it seems that incoming and outgoing requests are not traced automatically when I do this, so there is no trace on APM.

3. Sending the request and response to the library using a middleware, and adding `trace.id` manually to the logs.
This honestly seems too complicated, and even led to a memory leak in the code that maintains request and response using `async_hooks`.

Is there a simpler way to achieve this that I'm missing? I really want to integrate APM and Logs for better debugging, and I also don't want to remove this library and drop to using bare Pino.

Thanks.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.