elastic / elastic/elasticsearch
unable to easily use `ignore_malformed` in data streams
- Dominant language
- Java
- Stars
- 77.9k
- Forks
- 26.1k
- PR merge metrics
- PR metrics pending
Description
### Elasticsearch Version
8.10.0
### Installed Plugins
running from Kibana `yarn es snapshot`
### Java Version
_bundled_
### OS Version
Mac OSX latest
### Problem Description
We’re having difficulties using `ignore_malformed` at the index level in a data stream, as it seems like it really doesn’t want `@timestamp` to have that setting. This seems like an ES bug, since [the doc says](https://www.elastic.co/guide/en/elasticsearch/reference/current/ignore-malformed.html#ignore-malformed-setting)
> “The index.mapping.ignore_malformed setting can be set on the index level to ignore malformed content globally across all allowed mapping types. Mapping types that don’t support the setting will ignore it if set on the index level.“.
Technically, `@timestamp` is date, so not ignored because date types are “allowed”, but you can understand why @timestamp should not use that setting for data streams.
You also can't do the index-level and override just `@timestamp`, as it will complain that you can't use `ignore_malformed` on that field at all. So, in order for us to do what we want, we would have to add ignore_malformed: true on every field except @timestamp, which seems over-the-top hard.
We have the following issue open, blocked on resolution to this: https://github.com/elastic/kibana/issues/161465
### Steps to Reproduce
repro in Dev Console
```
PUT _index_template/pmuellr-index-template
{
"index_patterns": ["pmuellr-data-stream"],
"data_stream": { },
"template": {
"settings": {
"index.mapping.ignore_malformed": true
},
"mappings": {
"properties": {
"@timestamp": {
"type": "date"
},
"message": {
"type": "text"
}
}
}
}
}
{
"error": {
"root_cause": [
{
"type": "illegal_argument_exception",
"reason": "composable template [pmuellr-index-template] template after composition is invalid"
}
],
"type": "illegal_argument_exception",
"reason": "composable template [pmuellr-index-template] template after composition is invalid",
"caused_by": {
"type": "illegal_argument_exception",
"reason": "invalid composite mappings for [pmuellr-index-template]",
"caused_by": {
"type": "illegal_argument_exception",
"reason": "data stream timestamp field [@timestamp] has disallowed [ignore_malformed] attribute specified"
}
}
},
"status": 400
}
```
### Logs (if relevant)
_No response_
Contributor guide
Research direction
Start with the Dev Console reproduction using the composable index template and data stream settings shown in the issue. Follow the composite-mapping validation that rejects index.mapping.ignore_malformed for the @timestamp field, and define done as allowing the intended data-stream template configuration without weakening the timestamp field's constraints.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100