opensearch-project / opensearch-project/data-prepper
Unable to access JSON keys containing spaces using add_entries - value_expression, rename_keys, or copy_values processors
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 374
- Forks
- 354
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
We are encountering an issue in Data Prepper when trying to access or manipulate JSON fields whose key names contain spaces.
Example Input
{
"Computer Name": "xyz"
}
Attempt 1: add_entries with value_expression
- add_entries:
entries:
- key: cloud/provider
value_expression: "/Computer Name"
Observed behavior:
Using add_entries with value_expression results in a runtime error:
Error adding entry to record ... value_expression [/Computer Name]
When using add_entries with format instead of value_expression, no error is thrown, but the resulting value is null.
Attempt 2: rename_keys and copy_values
Both processors fail when referencing the same key with a space.
Observed errors:
java.lang.IllegalArgumentException:
key Computer Name must contain only alphanumeric chars with .-_@/
and must follow JsonPointer (ie. 'field/to/key')
And during pipeline validation:
Parameter "entries.null.from_key" for plugin "rename_keys" is invalid:
key Computer Name must contain only alphanumeric chars with .-_@/
and must follow JsonPointer (ie. 'field/to/key')
Expected Behavior
Data Prepper should either:
Support accessing JSON keys containing spaces using valid JSON Pointer syntax, or
Clearly document the limitation and provide a supported workaround for handling such keys (e.g., escaping, bracket notation, or preprocessing support).
Impact
This prevents ingestion and normalization of source data that contains valid JSON keys with spaces, which is common in several log and event sources.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the reported configurations for add_entries, rename_keys, and copy_values with the example JSON key "Computer Name". Compare the runtime and validation failures with the processors' supported key syntax, then establish whether valid JSON Pointer access or a documented workaround is the intended resolution.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100