splunk / splunk/docker-splunk

universalforwarder container fails to start when using k8s securityContext

Open
#537 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
549
Forks
277
Avg merge
3d 9h
Merged PRs (30d)
2

Description

I really hope I didn't miss anything obvious (I might have..), here is the problem.

Description

When using the securityContext as described here, the container for the universalforwarder will fail to start due to insufficient permissions. I am using universalforwarder version 8.1.

An example error message looks like this:
[0;31mtar: system/README/messages.conf.example: Cannot open: No such file or directory tar: system/README: Cannot mkdir: Permission denied

Expected behavior

The universalforwarder container should start without error.

Config files

These are the files I am currently running.

The universalforwarder Dockerfile is quite vanilla

FROM splunk/universalforwarder:8.1
COPY server.conf /opt/splunkforwarder/etc/system/local/server.conf
COPY deploymentclient.conf /opt/splunkforwarder/etc/system/local/deploymentclient.conf
COPY outputs.conf /opt/splunkforwarder/etc/apps/app/local/outputs.conf
COPY certs/ /mnt/certs/
EXPOSE 9997

Deployment.yaml (universalforwarder container):

- image: REDACTED
        imagePullPolicy: Always
        name: splunk-forwarder
        env:
          - name: SPLUNK_HOME_OWNERSHIP_ENFORCEMENT
            value: "false"
          - name: SPLUNK_PASSWORD
            valueFrom:
                secretKeyRef:
                  name: splunk-forwarder-local-user
                  key: password
          - name: SPLUNK_START_ARGS
            value: "--accept-license"
        securityContext:
          privileged: false
          allowPrivilegeEscalation: false
        ports:
          - name: tcp
            containerPort: 9997
            protocol: TCP

Deployment.yaml (spec.template.spec.securityContext):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: REDACTED
  namespace: REDACTED
spec:
  replicas: 1
  selector:
    #These labels are required for referencing other k8s objects
    matchLabels:
      app: "APP"
      servicegroup: "APPAPP"
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: "APP"
        servicegroup: "APPAPP"
      annotations:
    spec:
      securityContext:
          runAsUser: 41812
          fsGroup: 41812
          runAsNonRoot: true
          seccompProfile:
            type: RuntimeDefault

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the failure with the supplied Dockerfile and Kubernetes Deployment securityContext, using the universalforwarder:8.1 image. Inspect the container startup output around the tar permission errors. Done means the universalforwarder container starts successfully with runAsUser, fsGroup, runAsNonRoot, and disabled privilege escalation.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, kubernetes
Domain
devops, infrastructure
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.