elastic / elastic/beats

Filebeat and container metadata with containerd+nerdctl (no k8s)

Open
#43,539 2 comments 0 reactions 0 assignees View on GitHub
needs_team Stalled
Dominant language
Go
Stars
12.7k
Forks
5k
Avg merge
2d 15m
Merged PRs (30d)
385

Description

Hi!

We use containerd+nerdctl and compose schema. We have not so many apps and resources to use k8s. Now we want to ship logs to remote EFK so we can analyse them. Trouble is that we could not get container metadata with filebeat. `add_docker_metadata` dont work with containerd socket obviously, nerdctl place containers dirs only by there long id.

Is there any way to config log collecting with metadata from containerd?

In containers config we got this
```
{
...
"annotations": {
...
"com.docker.compose.project": "my-project",
"com.docker.compose.service": "my-app",
"nerdctl/name": "container-name",
...
}
....
}
```
Is it possible for filebeat to get it?

I think it could be useful for k8s users whenever they want to use filebeat for log collecting k8s and non-k8s machines and deploy filebeat to OS not in k8s.

Contributor guide

Open the contributing guide

Research direction

The issue names Filebeat's add_docker_metadata input, the containerd socket, and container annotations; start by tracing the metadata processor's supported runtime inputs. Confirm whether compose and nerdctl annotations can be exposed for non-Kubernetes containerd logs, and define completion as those fields appearing in shipped events with a documented configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
observability-sre
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.