Splunk UF - Warning: Attempting to revert the SPLUNK_HOME ownership causes troubles with Ansible play

HI,

when using the splunk/universalforwarder:9.2.1 image there are a lot of warnings about SPLUNK_HOME ownership when you run the image as user splunk as suggested in security.md

STDOUT:

Warning: Attempting to revert the SPLUNK_HOME ownership
Warning: Executing "chown -R splunk:splunk /opt/splunkforwarder"

Splunk> Needle. Haystack. Found.

Checking prerequisites...
        Checking mgmt port [8089]: open
        Checking conf files for problems...
        Done
        Checking default conf files for edits...
        Validating installed files against hashes from '/opt/splunkforwarder/splunkforwarder-9.2.1-78803f08aabb-linux-2.6-x86_64-manifest'
        All installed files intact.
        Done
All preliminary checks passed.

Starting splunk server daemon (splunkd)...  
Done


STDERR:

Couldn't change ownership for /opt/splunkforwarder/var: Operation not permitted
PYTHONHTTPSVERIFY is set to 0 in splunk-launch.conf disabling certificate validation for the httplib and urllib libraries shipped with the embedded Python interpreter; must be set to "1" for increased security

These are just warnings, so per se they do not cause any issues. In addition, this is listed as Known Issue SPL-226019 in the UF documentation.

However, in this play the warning turns into an problem: enable_admin_auth.yml#L6.

The initial splunk admin user setup processes stdout and here the warning results in a broken passwd file:

[splunk@splunk-uf-0 splunkforwarder]$ pwd
/opt/splunkforwarder
[splunk@splunk-uf-0 splunkforwarder]$ cat etc/passwd
:admin:Warning: Attempting to revert the SPLUNK_HOME ownership::administrator:admin:::19853

Therefore, all following plays that need the authentication fail (like setup of HEC).

As a workaround this works fine:

---
- name: Set admin access via seed
  when: first_run | bool
  block:
  - name: "Hash the password"
    command: "python -c 'import sys, crypt; print(crypt.crypt(sys.argv[1], crypt.mksalt(crypt.METHOD_SHA512)))' '{{ splunk.password }}'"
    register: hashed_pwd
    changed_when: hashed_pwd.rc == 0
    become: yes
    become_user: "{{ splunk.user }}"
    no_log: "{{ hide_password }}"

However, the root cause seems to be some underlaying issue with the permissions of Splunk.

Start by reproducing the splunk/universalforwarder:9.2.1 behavior under the splunk user and read docs/SECURITY.md alongside roles/splunk_common/tasks/enable_admin_auth.yml in the referenced Ansible project. Trace how the ownership warning reaches the initial admin setup and verify that authentication-dependent plays no longer receive a corrupted etc/passwd entry.