elastic / elastic/ecs-logging-java
Docker configuration
- Dominant language
- Java
- Stars
- 148
- Forks
- 82
- Avg merge
- 2d 17h
- Merged PRs (30d)
- 3
Description
Hi,
I am having a hard time working out how to use it in conjunction with docker. I have a Java application that logs everything in JSON to a file inside the docker container. I have created a `ln -sf /dev/stdout /opt/application.json` symlink, so everything will be output to stdout. Of course, looking into the -json.log file created by Docker in the `/var/lib/docker/container/containerid/*-json.log` it looks something like this:
```json
{"log":"{\"@timestamp\":\"2019-11-03T12:33:45.658Z\", \"log.level\": \"INFO\", \"message\":\"Login pkahr (philipp.kahr)\", \"service.name\":\"myservice\",\"process.thread.name\":\"http-bio-8080-exec-8\",\"log.logger\":\"com.myservice.server.core.utils.user.ThisDirectoryServiceImpl\",\"log.origin\":{\"file.name\":\"ThisDirectoryServiceImpl.java\",\"function\":\"loadUserByUsername\",\"file.line\":622}}\n","stream":"stdout","time":"2019-11-03T12:33:45.658546603Z"}
```
My filebeat config looks like this:
```yaml
filebeat.inputs:
- type: container
paths:
- '/var/lib/docker/containers/*/*.log'
json.keys_under_root: true
json.overwrite_keys: true
json.message_key: log
multiline.pattern: '^{'
multiline.negate: true
multiline.match: after
```
I am using the message decoder like suggested in the README
```yaml
- decode_json_fields:
fields: message
target: ""
overwrite_keys: true
# flattens the array to a single string
- script:
when:
has_fields: ['error.stack_trace']
lang: javascript
id: my_filter
source: >
function process(event) {
event.Put("error.stack_trace", event.Get("error.stack_trace").join("\n"));
}
```
however, I can see a couple of errors in `journalctl -u filebeat -f`
```
filebeat[29865]: 2019-11-03T13:56:09.252+0100 ERROR readjson/json.go:52 Error decoding JSON: json: cannot unmarshal number into Go value of type map[string]interface {}
filebeat[29865]: 2019-11-03T13:56:21.269+0100 ERROR readjson/json.go:52 Error decoding JSON: invalid character '-' in numeric literal
filebeat[29865]: 2019-11-03T13:56:21.269+0100 ERROR readjson/json.go:52 Error decoding JSON: invalid character 'E' looking for beginning of value
filebeat[29865]: 2019-11-03T13:56:21.268+0100 ERROR readjson/json.go:52 Error decoding JSON: unexpected EOF
filebeat[29865]: 2019-11-03T13:56:17.267+0100 WARN elasticsearch/client.go:535 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0x120f69c0, ext:63708382571, loc:(*time.Location)(nil)}, Meta:common.MapStr(nil), Fields:common.MapStr{"agent":common.MapStr{"ephemeral_id":"f3bb8324-dda5-4dbe-9f1a-6fc86820c37d", "hostname":"CentOS-76-64-minimal", "id":"d8cb4370-3918-40a8-88c5-80a992868234", "type":"filebeat", "version":"7.4.2"}, "ecs":common.MapStr{"version":"1.1.0"}, "host":common.MapStr{"architecture":"x86_64", "containerized":false, "hostname":"CentOS-76-64-minimal", "id":"7d91e31c18374a518bb44ad58646dfeb", "name":"CentOS-76-64-minimal", "os":common.MapStr{"codename":"Core", "family":"redhat", "kernel":"3.10.0-1062.4.1.el7.x86_64", "name":"CentOS Linux", "platform":"centos", "version":"7 (Core)"}}, "input":common.MapStr{"type":"container"}, "log":"", "log.level":"INFO", "log.logger":"com.myservice.server.core.utils.user.ServiceDirectoryServiceImpl", "log.origin":common.MapStr{"file.line":622, "file.name":"ServiceDirectoryServiceImpl.java", "function":"loadUserByUsername"}, "message":"Login pkahr (philipp.kahr)", "process.thread.name":"http-bio-8080-exec-2", "service.name":"myservice", "stream":"stdout", "tags":[]string{"testing"}}, Private:file.State{Id:"", Finished:false, Fileinfo:(*os.fileStat)(0xc00022cc30), Source:"/var/lib/docker/containers/b35f79cc3aa44c98709c3a05dfee5fcc13c92d18b0cff35d821af9820e52c398/b35f79cc3aa44c98709c3a05dfee5fcc13c92d18b0cff35d821af9820e52c398-json.log", Offset:656232, Timestamp:time.Time{wall:0xbf67d03f87d679e9, ext:12500845494, loc:(*time.Location)(0x4de6580)}, TTL:-1, Type:"container", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0x18802e6, Device:0x902}}, TimeSeries:false}, Flags:0x1} (status=400): {"type":"mapper_parsing_exception","reason":"object mapping for [log] tried to parse field [log] as object, but found a concrete value"}
```
Any idea on how to use the ECS logging in combination with docker? I would love to have it, so I can jump from the APM to the `container logs`.
Contributor guide
Research direction
Start with the Docker container input and message decoder configuration shown in the issue, then compare the nested Docker JSON with the ECS fields sent to Elasticsearch. Investigate the Filebeat JSON decoding errors and the mapping conflict on the log field. Done means identifying a configuration or logging adjustment that preserves the ECS fields without decoding or indexing errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, java
- Domain
- devops, observability-sre
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100