elastic / elastic/elasticsearch

unable to easily use `ignore_malformed` in data streams

Open
#98,511 8 comments 0 reactions 0 assignees View on GitHub
:StorageEngine/Data streams >bug Team:StorageEngine
Dominant language
Java
Stars
77.9k
Forks
26.1k
PR merge metrics
PR metrics pending

Description

### Elasticsearch Version

8.10.0

### Installed Plugins

running from Kibana `yarn es snapshot`

### Java Version

_bundled_

### OS Version

Mac OSX latest

### Problem Description

We’re having difficulties using `ignore_malformed` at the index level in a data stream, as it seems like it really doesn’t want `@timestamp` to have that setting. This seems like an ES bug, since [the doc says](https://www.elastic.co/guide/en/elasticsearch/reference/current/ignore-malformed.html#ignore-malformed-setting)
> “The index.mapping.ignore_malformed setting can be set on the index level to ignore malformed content globally across all allowed mapping types. Mapping types that don’t support the setting will ignore it if set on the index level.“.

Technically, `@timestamp` is date, so not ignored because date types are “allowed”, but you can understand why @timestamp should not use that setting for data streams.

You also can't do the index-level and override just `@timestamp`, as it will complain that you can't use `ignore_malformed` on that field at all. So, in order for us to do what we want, we would have to add ignore_malformed: true on every field except @timestamp, which seems over-the-top hard.

We have the following issue open, blocked on resolution to this: https://github.com/elastic/kibana/issues/161465

### Steps to Reproduce

repro in Dev Console

```
PUT _index_template/pmuellr-index-template
{
"index_patterns": ["pmuellr-data-stream"],
"data_stream": { },
"template": {
"settings": {
"index.mapping.ignore_malformed": true
},
"mappings": {
"properties": {
"@timestamp": {
"type": "date"
},
"message": {
"type": "text"
}
}
}
}
}

{
"error": {
"root_cause": [
{
"type": "illegal_argument_exception",
"reason": "composable template [pmuellr-index-template] template after composition is invalid"
}
],
"type": "illegal_argument_exception",
"reason": "composable template [pmuellr-index-template] template after composition is invalid",
"caused_by": {
"type": "illegal_argument_exception",
"reason": "invalid composite mappings for [pmuellr-index-template]",
"caused_by": {
"type": "illegal_argument_exception",
"reason": "data stream timestamp field [@timestamp] has disallowed [ignore_malformed] attribute specified"
}
}
},
"status": 400
}
```

### Logs (if relevant)

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with the Dev Console reproduction using the composable index template and data stream settings shown in the issue. Follow the composite-mapping validation that rejects index.mapping.ignore_malformed for the @timestamp field, and define done as allowing the intended data-stream template configuration without weakening the timestamp field's constraints.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.