Graylog2 / Graylog2/graylog2-server
Dots in field names are replaced silently
- Dominant language
- Java
- Stars
- 8.1k
- Forks
- 1.1k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 217
Description
The fact that we don't support dots in field names is nothing new.
This code has been there since ages, silently replacing dots with underscores once a Message gets written to ES/OS:
https://github.com/Graylog2/graylog2-server/blob/master/graylog2-server/src/main/java/org/graylog2/plugin/Message.java#L404-L406
What is problematic, is that we neither have documentation for this behavior, nor do we log a warning if we replace dots.
The fact that field renaming happens very late in the processing can lead to some confusion when writing pipeline rules or extractors.
The ingested Message will show up with underscore fields in the search, but within processing you need to work with dots.
## Possible solutions
- Add documentation
- Show a system notification if we detect that changing field names was needed. (This needs to be done efficiently to not cause a performance degradation)
- ~Change the field names directly when assigning them~ (This will break existing rules and might confuse users who are already accustomed to the existing behavior)
## Related problems
Slash characters in field names are a similar case. They are not allowed in ES/OS. In this particular case we behave differently and ignore the field containing slash in its name, while ingesting the rest of the message. The problem has been mentioned in #12990, where we have introduced rate limited logging to inform about dropped fields with INFO level.
It seems that we may need a single issue to solve all the problems related with special characters in field names, and this issue has been chosen for that.
Refs:
https://github.com/Graylog2/graylog2-server/issues/12990
https://github.com/Graylog2/graylog2-server/issues/6588
https://github.com/Graylog2/graylog2-server/pull/5983
https://github.com/Graylog2/graylog2-server/issues/4583
https://github.com/elastic/elasticsearch/issues/15951
Contributor guide
Assessment
This issue has not been assessed yet.