x-pack/filebeat/input/httpjson: `split[].keep_parent` overrides `split[].ignore_empty_value`
- Dominant language
- Go
- Stars
- 12.7k
- Forks
- 5k
- Avg merge
- 2d 15m
- Merged PRs (30d)
- 385
Description
For the HTTP JSON input, the following split config will return the root as an event for an incoming document with no `items` key or `events` key.
```yaml
response.split:
target: body.items
ignore_empty_value: true
split:
target: body.events
keep_parent: true
```
This may be unexpected and results in bugs such as those fixed in https://github.com/elastic/integrations/pull/15948. Other potential cases are listed in [this comment](https://github.com/elastic/integrations/pull/15948#issuecomment-3540095055).
It works this way because `ignore_empty_value` does not stop the nested split operation, as explained in the documentation:
> [`response.split[].ignore_empty_value`](https://www.elastic.co/docs/reference/beats/filebeat/filebeat-input-httpjson#_response_split_ignore_empty_value)
>
> If set to true, empty or missing value will be ignored and processing will pass on to the next nested split operation instead of failing with an error. Default: `false`.
> ...
If the nested split operation had set `ignore_empty_value: true`, the document root would still be returned as an event, because it also sets `keep_parent: true`.
This behavior is not made clear in the documentation for [`response.split`](https://www.elastic.co/docs/reference/beats/filebeat/filebeat-input-httpjson#response-split), [`response.split[].keep_parent`](https://www.elastic.co/docs/reference/beats/filebeat/filebeat-input-httpjson#_response_split_keep_parent) or [`response.split[].ignore_empty_value`](https://www.elastic.co/docs/reference/beats/filebeat/filebeat-input-httpjson#_response_split_ignore_empty_value), but it was added intentionally, in https://github.com/elastic/beats/pull/34322.
Changing that behavior makes [these tests](https://github.com/elastic/beats/blob/3b8b99673e633cb4f4240d12d19709b82e9fa6cb/x-pack/filebeat/input/httpjson/input_test.go#L179-L238) fail:
`TestInput/split_on_null_field_with_ignore_empty_value_keeping_parent`
`TestInput/split_on_empty_array_with_ignore_empty_value_keeping_parent`
`TestInput/split_on_null_field_at_root_with_ignore_empty_value_keeping_parent`
`TestInput/split_on_empty_array_at_root_with_ignore_empty_value_keeping_parent`
It would require breaking changes to untangle the options for splitting, repeating the parent context for a split, and using the root document as a fallback if the split target is empty.
Without breaking changes it may be possible to explain the current behavior more clearly in the documentation.
### Related issues
- Related https://github.com/elastic/integrations/pull/15948
- Related https://github.com/elastic/integrations/pull/9974
Contributor guide
Assessment
This issue has not been assessed yet.