Add merge_arrays processor and enhance convert_entry_type for nested keys in iterate_on

[FEATURE] Add merge_arrays processor and enhance convert_entry_type for nested keys in iterate_on

Summary

We are building OCSF integrations for ExtraHop Reveal(x) 360 detection events and have encountered limitations in existing processors when performing complex parallel-array merging and nested-key type conversions within array elements.

Several real-world integration use cases require:

• Merging two parallel arrays into a single enriched array by matching index positions
• Converting types of nested fields within array elements (e.g., src_endpoint/uid inside each element of a participants array)
• Supporting fallback field resolution when the primary source field is null or empty

These capabilities are currently not supported by existing processors such as add_entry, copy_values, rename_keys, list_to_map, or convert_entry_type.

For this we need:

1. A new merge_arrays processor to combine parallel arrays by index
2. Enhancement to convert_entry_type to support slash-delimited nested keys when used with iterate_on

---

Use Case 1 – Merging Parallel MITRE ATT&CK Arrays (ExtraHop Reveal(x) 360)

Problem Description

ExtraHop detection events contain MITRE ATT&CK data as two separate parallel arrays — mitre_tactics and mitre_techniques — where elements at the same index are related. OCSF Detection Finding (class 2004) requires these to be combined into a single finding_info.attacks[] array where each element contains both the tactic and its corresponding technique as nested objects.

Currently, there is no processor that can merge fields from one array into another by matching index positions. The only workaround is to hardcode temporary variables per index (e.g., _tmp_atk_0, _tmp_atk_1), which:

• Limits support to a fixed number of elements
• Creates verbose, error-prone mapping configurations
• Does not scale for events with variable-length arrays

Sample Input

{
  "mitre_tactics": [
    {"id": "TA0001", "name": "Initial Access", "url": "https://attack.mitre.org/tactics/TA0001"},
    {"id": "TA0006", "name": "Credential Access", "url": "https://attack.mitre.org/tactics/TA0006"}
  ],
  "mitre_techniques": [
    {"id": "T1040", "name": "Network Sniffing", "url": "https://attack.mitre.org/techniques/T1040", "legacy_ids": ["T1040"]},
    {"id": "T1078", "name": "Valid Accounts", "url": "https://attack.mitre.org/techniques/T1078", "legacy_ids": ["T1078"]}
  ]
}

Expected Output

After restructuring tactics in-place (using existing add_entries with iterate_on) and then merging techniques by index:

{
  "finding_info": {
    "attacks": [
      {
        "tactic": {"uid": "TA0001", "name": "Initial Access", "src_url": "https://attack.mitre.org/tactics/TA0001"},
        "technique": {"uid": "T1040", "name": "Network Sniffing", "src_url": "https://attack.mitre.org/techniques/T1040"}
      },
      {
        "tactic": {"uid": "TA0006", "name": "Credential Access", "src_url": "https://attack.mitre.org/tactics/TA0006"},
        "technique": {"uid": "T1078", "name": "Valid Accounts", "src_url": "https://attack.mitre.org/techniques/T1078"}
      }
    ]
  }
}

Proposed Configuration

- merge_arrays:
    source: mitre_techniques
    target: mitre_tactics
    entries:
      - from_key: id
        to_key: technique/uid
        fallback_key: legacy_ids/0
      - from_key: name
        to_key: technique/name
      - from_key: url
        to_key: technique/src_url
    merge_arrays_when: '/mitre_techniques/0 != null'

---

Use Case 2 – Converting Nested Fields Within Array Elements

Problem Description

When transforming ExtraHop participants[] into OCSF evidences[], each participant element is restructured to contain nested endpoint objects like src_endpoint or dst_endpoint. After restructuring, we need to convert the type of nested fields such as src_endpoint/uid (from integer to string) within each array element.

Currently, convert_entry_type with iterate_on only supports flat keys within array elements. For example:

# This works — flat key
- convert_entry_type:
    key: uid
    type: string
    iterate_on: participants

# This does NOT work — nested key
- convert_entry_type:
    key: src_endpoint/uid
    type: string
    iterate_on: participants

The nested key src_endpoint/uid is not resolved within each element; the processor attempts a flat item.get("src_endpoint/uid") which returns null.

Sample Input

{
  "participants": [
    {"src_endpoint": {"ip": "10.0.0.1", "uid": 4294976402}},
    {"dst_endpoint": {"ip": "10.0.0.2", "uid": 100002}},
    {"src_endpoint": {"ip": "10.0.0.5", "uid": 100005}}
  ]
}

Expected Output

{
  "participants": [
    {"src_endpoint": {"ip": "10.0.0.1", "uid": "4294976402"}},
    {"dst_endpoint": {"ip": "10.0.0.2", "uid": 100002}},
    {"src_endpoint": {"ip": "10.0.0.5", "uid": "100005"}}
  ]
}

Note: Only src_endpoint/uid is converted to string; elements without src_endpoint are left unchanged. dst_endpoint/uid remains an integer because the key path doesn't match.

Proposed Configuration

- convert_entry_type:
    key: src_endpoint/uid
    type: string
    iterate_on: participants

---

Use Case 3 – Merging Participant Data with Root-Level Properties

Problem Description

ExtraHop detection events contain port information at the root level in properties.client_port and properties.server_port, while participant/endpoint data is in a separate participants[] array. We need to enrich specific array elements (client endpoints) with root-level port values.

While add_entries with iterate_on and evaluate_when_on_element: true can conditionally add fields to array elements, the merge_arrays processor provides a cleaner pattern when the enrichment source is a parallel array. For root-level field injection into array elements, the existing add_entries with disable_root_keys: false works but the merge_arrays processor complements this for array-to-array scenarios.

---

Limitations with Existing Processors

We evaluated add_entry, copy_values, rename_keys, list_to_map, map_to_list, and convert_entry_type. Below are the observed limitations:

1. No Index-Based Array Merging

• No existing processor can merge fields from one array into another by matching index positions.
• list_to_map converts arrays to maps using a key field, which doesn't preserve index-based relationships and would overwrite duplicate keys.
• copy_values operates on the entire field, not element-by-element.
• The only workaround is hardcoding temporary variables per index (_tmp_atk_0, _tmp_atk_1, etc.), which doesn't scale.

2. No Nested Key Support in convert_entry_type with iterate_on

• When iterate_on is specified, convert_entry_type uses item.get(key) and item.put(key, value) directly on each element map.
• This only works for flat keys (e.g., uid), not slash-delimited nested paths (e.g., src_endpoint/uid).
• The path traversal logic that exists for top-level event keys is not applied within iterate_on element maps.

3. No Fallback Key Resolution

• When merging fields between arrays, the primary source field may be null or empty in some elements (e.g., id is null but legacy_ids/0 has the value).
• No existing processor supports a fallback field path when the primary source resolves to null.

---

Proposed Solution

New merge_arrays Processor

A processor that merges elements from a source array into a target array by matching index positions:

Configuration:

Parameter	Required	Description
source	Yes	Key of the source array
target	Yes	Key of the target array
entries	Yes	List of field mappings with from_key, to_key, and optional fallback_key
merge_arrays_when	No	Conditional expression for whether the processor runs

Behavior:

• Iterates min(source.size, target.size) elements
• For each index, copies specified fields from source element to target element
• Supports slash-delimited nested paths for both from_key and to_key (e.g., legacy_ids/0, technique/uid)
• Creates intermediate maps automatically for nested to_key paths
• Optional fallback_key used when from_key resolves to null or empty string
• Gracefully skips null elements and non-Map elements with a warning log
• Supports list index access in from_key (e.g., legacy_ids/0)

Enhanced convert_entry_type with Nested Key Support for iterate_on

When iterate_on is specified and the key contains /, the processor should:

1. Split the key path once before the loop (avoid recomputing key.split("/") per element)
2. Walk the nested map path to resolve the parent map and leaf key
3. Read the value, convert it, and write it back — all in a single traversal
4. Gracefully handle missing intermediate paths (skip the element, don't fail)

---

Why This Matters

These requirements are not integration-specific edge cases. Parallel array merging and nested-key type conversion within arrays occur across multiple OCSF integrations:

• MITRE ATT&CK mapping: Any source that provides tactics and techniques as separate arrays (ExtraHop, CrowdStrike, SentinelOne, etc.)
• Participant/endpoint enrichment: Any source with participant arrays containing nested endpoint objects that need type normalization
• Evidence construction: Building OCSF evidences[] from source arrays with nested src_endpoint, dst_endpoint, user, and actor objects

Without these capabilities, mapping configurations require verbose index-hardcoded workarounds that don't scale and are error-prone.

---

Conclusion

Due to the above limitations, we are currently unable to:

1. Correctly merge parallel MITRE ATT&CK arrays into OCSF attacks[] for variable-length inputs
2. Convert types of nested fields within array elements after restructuring

A new merge_arrays processor and enhancement to convert_entry_type would significantly improve Data Prepper's flexibility for real-world security integrations involving complex array transformations.

We would appreciate feedback from maintainers on:

• Whether this fits within the roadmap
• Preferred design direction (new processor vs. enhancement to existing)
• Contribution guidelines if we plan to implement this enhancement

We have a working implementation with full test coverage that we can contribute as a PR.

Start by reviewing the existing add_entry, copy_values, rename_keys, list_to_map, map_to_list, and convert_entry_type processors, focusing on their iterate_on behavior. Compare them with the proposed merge_arrays configuration and the stated working implementation with full test coverage. Done means maintainers agree on the design and both array merging and nested-key conversion are implemented with tests.