opensearch-project / opensearch-project/data-prepper
[BUG] Delete entries and any processor that uses delete does not work when field name has '/'
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 374
- Forks
- 354
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
Describe the bug
When leaf field in a field path to delete has a '/' (replaced by "~1") in the processor config, the field is not deleted. Further renaming also does not work as only the value is copied but the original field is not deleted.
To Reproduce
Input Log:
{
"A/B": "allow"
}
Delete configuration
processor:
- delete_entries:
with_keys: ["A~1B"]
does not delete the key (neither does replacing "A~1B" with "A/B").
Expected behavior
The field should be deleted.
Screenshots
NA
Environment (please complete the following information):
- OS: MacOS 15.5 (24F74)
Additional context
From what I see this is likely an issue with the delete method in JacksonEvent. Specifically this line https://github.com/opensearch-project/data-prepper/blob/main/data-prepper-api/src/main/java/org/opensearch/dataprepper/model/event/JacksonEvent.java#L392 where we delete using ObjectNode which expects the field name to be the exact field name (i.e. without anything escaped like "~1"). This means that leafKey has to be "A/B" not "A~1B" . We still need '/' to be escaped because in https://github.com/opensearch-project/data-prepper/blob/main/data-prepper-api/src/main/java/org/opensearch/dataprepper/model/event/JacksonEvent.java#L380 we split by '/' (the separator for nesting) and make a jsonpointer out of the base and retrieve the basenode https://github.com/opensearch-project/data-prepper/blob/main/data-prepper-api/src/main/java/org/opensearch/dataprepper/model/event/JacksonEvent.java#L386.
To get the equivalent string field name of a json pointer, we can use getMatchingProperty method, so replacing the line https://github.com/opensearch-project/data-prepper/blob/main/data-prepper-api/src/main/java/org/opensearch/dataprepper/model/event/JacksonEvent.java#L392 to
((ObjectNode) baseNode).remove(toJsonPointer(leafKey).getMatchingProperty());
Achieves the desired behaviour.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in data-prepper-api/src/main/java/org/opensearch/dataprepper/model/event/JacksonEvent.java around lines 380-392, then trace the delete_entries processor path for a field named A/B. Add regression coverage for deleting that field and verify that delete-based renaming removes the original field as well as copying its value.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100