elastic / elastic/ecs-logging-java
Docker configuration
- Langage dominant
- Java
- Étoiles
- 148
- Forks
- 82
- Merge moyen
- 2 j 17 h
- PR mergées (30 j)
- 3
Description
Hi,
I am having a hard time working out how to use it in conjunction with docker. I have a Java application that logs everything in JSON to a file inside the docker container. I have created a `ln -sf /dev/stdout /opt/application.json` symlink, so everything will be output to stdout. Of course, looking into the -json.log file created by Docker in the `/var/lib/docker/container/containerid/*-json.log` it looks something like this:
```json
{"log":"{\"@timestamp\":\"2019-11-03T12:33:45.658Z\", \"log.level\": \"INFO\", \"message\":\"Login pkahr (philipp.kahr)\", \"service.name\":\"myservice\",\"process.thread.name\":\"http-bio-8080-exec-8\",\"log.logger\":\"com.myservice.server.core.utils.user.ThisDirectoryServiceImpl\",\"log.origin\":{\"file.name\":\"ThisDirectoryServiceImpl.java\",\"function\":\"loadUserByUsername\",\"file.line\":622}}\n","stream":"stdout","time":"2019-11-03T12:33:45.658546603Z"}
```
My filebeat config looks like this:
```yaml
filebeat.inputs:
- type: container
paths:
- '/var/lib/docker/containers/*/*.log'
json.keys_under_root: true
json.overwrite_keys: true
json.message_key: log
multiline.pattern: '^{'
multiline.negate: true
multiline.match: after
```
I am using the message decoder like suggested in the README
```yaml
- decode_json_fields:
fields: message
target: ""
overwrite_keys: true
# flattens the array to a single string
- script:
when:
has_fields: ['error.stack_trace']
lang: javascript
id: my_filter
source: >
function process(event) {
event.Put("error.stack_trace", event.Get("error.stack_trace").join("\n"));
}
```
however, I can see a couple of errors in `journalctl -u filebeat -f`
```
filebeat[29865]: 2019-11-03T13:56:09.252+0100 ERROR readjson/json.go:52 Error decoding JSON: json: cannot unmarshal number into Go value of type map[string]interface {}
filebeat[29865]: 2019-11-03T13:56:21.269+0100 ERROR readjson/json.go:52 Error decoding JSON: invalid character '-' in numeric literal
filebeat[29865]: 2019-11-03T13:56:21.269+0100 ERROR readjson/json.go:52 Error decoding JSON: invalid character 'E' looking for beginning of value
filebeat[29865]: 2019-11-03T13:56:21.268+0100 ERROR readjson/json.go:52 Error decoding JSON: unexpected EOF
filebeat[29865]: 2019-11-03T13:56:17.267+0100 WARN elasticsearch/client.go:535 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0x120f69c0, ext:63708382571, loc:(*time.Location)(nil)}, Meta:common.MapStr(nil), Fields:common.MapStr{"agent":common.MapStr{"ephemeral_id":"f3bb8324-dda5-4dbe-9f1a-6fc86820c37d", "hostname":"CentOS-76-64-minimal", "id":"d8cb4370-3918-40a8-88c5-80a992868234", "type":"filebeat", "version":"7.4.2"}, "ecs":common.MapStr{"version":"1.1.0"}, "host":common.MapStr{"architecture":"x86_64", "containerized":false, "hostname":"CentOS-76-64-minimal", "id":"7d91e31c18374a518bb44ad58646dfeb", "name":"CentOS-76-64-minimal", "os":common.MapStr{"codename":"Core", "family":"redhat", "kernel":"3.10.0-1062.4.1.el7.x86_64", "name":"CentOS Linux", "platform":"centos", "version":"7 (Core)"}}, "input":common.MapStr{"type":"container"}, "log":"", "log.level":"INFO", "log.logger":"com.myservice.server.core.utils.user.ServiceDirectoryServiceImpl", "log.origin":common.MapStr{"file.line":622, "file.name":"ServiceDirectoryServiceImpl.java", "function":"loadUserByUsername"}, "message":"Login pkahr (philipp.kahr)", "process.thread.name":"http-bio-8080-exec-2", "service.name":"myservice", "stream":"stdout", "tags":[]string{"testing"}}, Private:file.State{Id:"", Finished:false, Fileinfo:(*os.fileStat)(0xc00022cc30), Source:"/var/lib/docker/containers/b35f79cc3aa44c98709c3a05dfee5fcc13c92d18b0cff35d821af9820e52c398/b35f79cc3aa44c98709c3a05dfee5fcc13c92d18b0cff35d821af9820e52c398-json.log", Offset:656232, Timestamp:time.Time{wall:0xbf67d03f87d679e9, ext:12500845494, loc:(*time.Location)(0x4de6580)}, TTL:-1, Type:"container", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0x18802e6, Device:0x902}}, TimeSeries:false}, Flags:0x1} (status=400): {"type":"mapper_parsing_exception","reason":"object mapping for [log] tried to parse field [log] as object, but found a concrete value"}
```
Any idea on how to use the ECS logging in combination with docker? I would love to have it, so I can jump from the APM to the `container logs`.
Guide de contribution
Ouvrir le guide de contribution
Piste de recherche
Commencez par la configuration de l’input du conteneur Docker et du décodeur de messages présentée dans l’issue, puis comparez le JSON Docker imbriqué avec les champs ECS envoyés à Elasticsearch. Examinez les erreurs de décodage JSON de Filebeat et le conflit de mapping sur le champ log. La tâche est terminée lorsqu’un ajustement de configuration ou de logging qui préserve les champs ECS sans erreurs de décodage ni d’indexation a été identifié.
Rédigé par le modèle d'indexation à partir du texte de l'issue.
Évaluation
- Stack technique
- docker, java
- Domaine
- devops, observability-sre
- Type d'issue
- Bug
- Difficulté
- 4/5
- Temps estimé
- 3-5 jours
- Activité
- À l'abandon
- Clarté
- À clarifier
- Accessibilité débutants
- 25/100