opensearch-project / opensearch-project/data-prepper

Unable to access JSON keys containing spaces using add_entries - value_expression, rename_keys, or copy_values processors

Open
#6,503 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement help wanted
Dominant language
Java
Stars
374
Forks
354
Avg merge
3d 18h
Merged PRs (30d)
8

Description

We are encountering an issue in Data Prepper when trying to access or manipulate JSON fields whose key names contain spaces.

Example Input
{
      "Computer Name": "xyz"
}
Attempt 1: add_entries with value_expression
- add_entries:
    entries:
      - key: cloud/provider
        value_expression: "/Computer Name"
Observed behavior:

Using add_entries with value_expression results in a runtime error:

Error adding entry to record ... value_expression [/Computer Name]

When using add_entries with format instead of value_expression, no error is thrown, but the resulting value is null.

Attempt 2: rename_keys and copy_values

Both processors fail when referencing the same key with a space.

Observed errors:

java.lang.IllegalArgumentException:
key Computer Name must contain only alphanumeric chars with .-_@/
and must follow JsonPointer (ie. 'field/to/key')

And during pipeline validation:

Parameter "entries.null.from_key" for plugin "rename_keys" is invalid:
key Computer Name must contain only alphanumeric chars with .-_@/
and must follow JsonPointer (ie. 'field/to/key')

Expected Behavior

Data Prepper should either:

Support accessing JSON keys containing spaces using valid JSON Pointer syntax, or

Clearly document the limitation and provide a supported workaround for handling such keys (e.g., escaping, bracket notation, or preprocessing support).

Impact

This prevents ingestion and normalization of source data that contains valid JSON keys with spaces, which is common in several log and event sources.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the reported configurations for add_entries, rename_keys, and copy_values with the example JSON key "Computer Name". Compare the runtime and validation failures with the processors' supported key syntax, then establish whether valid JSON Pointer access or a documented workaround is the intended resolution.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.