splunk / splunk/security_content

Lookup replication issue with DA-ESS-ContentUpdate on distributed search

Open
#4,229 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.7k
Forks
494
Avg merge
1d 14h
Merged PRs (30d)
25

Description

Hi,

I am facing an issue with one of the lookups provided by DA-ESS-ContentUpdate in a distributed Splunk Enterprise Security environment.

Environment
Splunk Enterprise: 10.4
Splunk Enterprise Security: 8.x
Search Head Cluster: 3 members
Multiple Indexers
Linux
The lookup is a CSV-based lookup provided by DA-ESS-ContentUpdate

The lookup is available and works correctly on the Search Head, but it is not available on the Search Peers/Indexers when the search is distributed.

Lookup

The lookup definition and CSV are both located under:

/opt/splunk/etc/apps/DA-ESS-ContentUpdate/

The lookup CSV is under:

/opt/splunk/etc/apps/DA-ESS-ContentUpdate/lookups/

and its definition is in the corresponding transforms.conf.

The lookup works correctly when executed locally on the Search Head, but distributed searches fail because the Search Peer does not have the lookup.

Investigation

We checked the effective distsearch.conf configuration using:

/opt/splunk/bin/splunk btool distsearch list replicationDenylist --debug

The following entry was found:

/opt/splunk/etc/apps/DA-ESS-ContentUpdate/default/distsearch.conf
[replicationDenylist]
excludeESCU = apps[/\]DA-ESS-ContentUpdate[/\]lookups[/\]...

This appears to exclude all lookup files under DA-ESS-ContentUpdate/lookups/ from Knowledge Bundle replication.

At the same time, the global configuration contains:

replicate.lookups = true

and the effective replication allowlist contains:

[replicationAllowlist]
other = (system|(apps/(?!pdfserver))|users(/_reserved)?//*)/(bin|lookups)/...

Therefore, lookups are generally allowed to be replicated, but the ESCU-specific denylist appears to explicitly exclude the entire DA-ESS-ContentUpdate/lookups/ directory.

We also checked:

/opt/splunk/bin/splunk btool distsearch list replicationSettings:refineConf --debug

which confirms:

replicate.lookups = true
replicate.transforms = true

However, the replicationDenylist entry above prevents the ESCU lookup from being included in the Knowledge Bundle.

Knowledge Bundle verification

We inspected the generated Knowledge Bundle and confirmed that the relevant lookup is not present in the bundle.

This explains why:

The lookup exists on the Search Head.
The lookup definition exists.
replicate.lookups = true.
The lookup still does not exist on the Search Peer.
Distributed searches cannot use the lookup.
Question

Is this exclusion intentional for ESCU lookups?

If so, what is the recommended/supported way to replicate a specific ESCU lookup to Search Peers while keeping the other ESCU lookup files excluded?

We specifically want to achieve the following:

DA-ESS-ContentUpdate/lookups/

lookup-A.csv                         -> NOT replicated
lookup-B.csv                         -> NOT replicated
...
cisco_secure_firewall_filetype_lookup.csv
                                      -> SHOULD be replicated

We do not want to replicate all ESCU lookups because of the potential Knowledge Bundle size and performance impact.

We also do not want to copy the lookup into a custom App, because doing so would cause the custom copy to become independent from future ESCU Content Updates. If Splunk updates this lookup in a future Content Update, we would want the updated official lookup to continue being used automatically.

Expected behavior

Ideally, we would like to keep the lookup managed by:

DA-ESS-ContentUpdate

so that it continues to receive future Content Updates, while allowing only this specific lookup to be included in Knowledge Bundle replication.

Is there a supported configuration or recommended approach for creating an exception to:

excludeESCU = apps[/\]DA-ESS-ContentUpdate[/\]lookups[/\]...

for a single lookup?

Thanks.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with DA-ESS-ContentUpdate/default/distsearch.conf and the excludeESCU entry, then run the two splunk btool commands named in the issue. Verify the generated Knowledge Bundle against the lookup path and determine whether a supported per-file exception exists without copying the lookup. Done means documenting the supported configuration or confirming that no such exception is available.

Written by the indexing model from the issue text.

Assessment

Domain
backend, distributed-systems, search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.