prometheus / prometheus/alertmanager

Alertmanager cannot access its own data with latest Docker image

Open
#1,740 9 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

kind/documentation
Dominant language
Go
Stars
8.6k
Forks
2.5k
Avg merge
2d 6h
Merged PRs (30d)
61

Description

I upgraded to the latest Docker image (from unknown earlier version, maybe 6 months old) and now discover that alertmanager cannot access its own data anymore.

level=error ts=2019-02-05T10:08:55.171604943Z caller=main.go:221 err="open /alertmanager/nflog: permission denied"

The config and data are stored in a directory owned by root:root with only root having access. The data directory is mapped to /alertmanager/ in the container. The container is configured to run without any user account being explicitly specified, whatever that means ("use what is in the dockerfile, default root" I believe).

From host, the permissions of the directories are:

drwxrwx---  2 root root 4096 Feb  5 10:57 config
drwxrwx---  2 root root 4096 Feb  5 10:40 data

And the contents of the data directory are:

-rw-rw---- 1 root root  156 Feb  5 10:40 nflog
-rw-rw---- 1 root root  501 Feb  5 10:40 silences

Possibly related to #1585 and #1586?

I am not a Linux access control expert but I consider it VERY dubious to do any permission/user management inside a container. How permissions are assigned to mapped volumes and their contents and what user account a container runs as are decisions for the operator who runs the container and the software inside a container should not be doing any touching of permissions or user accounts at all. Alertmanager is not the only thing using the filesystem and it could not possibly know in advance what security configuration I desire on my server's filesystem.

If this is some intentional feature, please describe the exact usage pattern that allows me to provide root:root owned config and data directories to the Alertmanager container. Right now I cannot determine a correct way to start alertmanager without sharing my data directories excessively to non-root users.

I am open to adjusting my filesystem setup. My goal is to have the config and data accessible only to Alertmanager and root/sudoers. This used to work if I just had my files restricted to root:root but not anymore, apparently.

I can work around this by adding -u root to my Docker run statement. I guess it overrides the configuration in the Dockerfile to some extent? Seems like a hack that should not be needed, though. Please provide a mechanism that does not require such an action. If you think this is the right solution, please document it. Doing a chown seems wrong in any case - the container does not own my filesystem.

(Yes, the same problem affects latest Prometheus version)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the latest Docker image with root-owned, restricted host directories mounted at /alertmanager, using the Docker run configuration described in the issue. Check the image's configured user and permission behavior; done means Alertmanager can access its data without requiring -u root, or the correct usage is documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.