elastic / elastic/beats

x-pack/filebeat/input/httpjson: `split[].keep_parent` overrides `split[].ignore_empty_value`

Open
#47,699 2 comments 0 reactions 1 assignee Claimed by @chrisberkhout View on GitHub
bug Filebeat input:HTTPJSON Team:Security-Service Integrations
Dominant language
Go
Stars
12.7k
Forks
5k
Avg merge
2d 15m
Merged PRs (30d)
385

Description

For the HTTP JSON input, the following split config will return the root as an event for an incoming document with no `items` key or `events` key.
```yaml
response.split:
target: body.items
ignore_empty_value: true
split:
target: body.events
keep_parent: true
```
This may be unexpected and results in bugs such as those fixed in https://github.com/elastic/integrations/pull/15948. Other potential cases are listed in [this comment](https://github.com/elastic/integrations/pull/15948#issuecomment-3540095055).

It works this way because `ignore_empty_value` does not stop the nested split operation, as explained in the documentation:

> [`response.split[].ignore_empty_value`](https://www.elastic.co/docs/reference/beats/filebeat/filebeat-input-httpjson#_response_split_ignore_empty_value)
>
> If set to true, empty or missing value will be ignored and processing will pass on to the next nested split operation instead of failing with an error. Default: `false`.
> ...

If the nested split operation had set `ignore_empty_value: true`, the document root would still be returned as an event, because it also sets `keep_parent: true`.

This behavior is not made clear in the documentation for [`response.split`](https://www.elastic.co/docs/reference/beats/filebeat/filebeat-input-httpjson#response-split), [`response.split[].keep_parent`](https://www.elastic.co/docs/reference/beats/filebeat/filebeat-input-httpjson#_response_split_keep_parent) or [`response.split[].ignore_empty_value`](https://www.elastic.co/docs/reference/beats/filebeat/filebeat-input-httpjson#_response_split_ignore_empty_value), but it was added intentionally, in https://github.com/elastic/beats/pull/34322.

Changing that behavior makes [these tests](https://github.com/elastic/beats/blob/3b8b99673e633cb4f4240d12d19709b82e9fa6cb/x-pack/filebeat/input/httpjson/input_test.go#L179-L238) fail:
`TestInput/split_on_null_field_with_ignore_empty_value_keeping_parent`
`TestInput/split_on_empty_array_with_ignore_empty_value_keeping_parent`
`TestInput/split_on_null_field_at_root_with_ignore_empty_value_keeping_parent`
`TestInput/split_on_empty_array_at_root_with_ignore_empty_value_keeping_parent`

It would require breaking changes to untangle the options for splitting, repeating the parent context for a split, and using the root document as a fallback if the split target is empty.

Without breaking changes it may be possible to explain the current behavior more clearly in the documentation.

### Related issues

- Related https://github.com/elastic/integrations/pull/15948
- Related https://github.com/elastic/integrations/pull/9974

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.