Splunk Search Heads fail to start in Docker Swarm

Issue Description:

So I have been experimenting with docker swarm and run into an issue where splunk containers with role: search_head or search_head captain fail to start in a docker swarm environment.

Project Codebase:

https://github.com/rskntroot/splunk

NOTE: I understand that splunk in docker swarm is unsupported for a reason.

NOTE: I have managed to get a 3x [search_head] 1x [deployer] 1x [indexer] setup to work fully in docker swarm with the following workaround

Work Around: **

• [deployer] and [indexer] were configured with environment variables & defaults.yml
• [search_head] containers were deployed without a role and then manually configured for shc

During testing of the workaround I have found that:

• search functionality, working as expected.
• app deployment, working as expected.
• artifact replication, working as expected.
  Issues with workaround:
• Splunk search head configuration does not persist in the event the docker container fails (it will be rebuilt with no role)

Conclusion:

Pre workaround: I was able to docker exec into a search container and unable to connect to the other search nodes. No issues with connecting to the deployer or indexer.

It seems that the splunk-ansible configurations do not put the container in a state where docker swarm will publish the containers IP to docker DNS.

I'm at wits-end on this one and was wondering if anyone wants to give me some pointers on how to create an ansible playbook for this case 🤷🏻‍♂️ (setting docker state is handled in entrypoint.sh)

Start by reviewing entrypoint.sh, defaults.yml, and the splunk-ansible configuration involved when search_head or search_head captain containers start in Docker Swarm. Compare the configured and manually configured deployments, then determine what change would preserve search head configuration and allow the search nodes to connect through Docker DNS.