Nginx container consumes much more memory than expected

Setup

We run docker.io/nginx:1.15.8 containers with simple proxy + disk cache configuration:

events {
  worker_connections 2048;
}

http {
  # Similar to default "combined" log format, appends additional info at the end.
  # - request_time: time it took for Nginx to handle request.
  # - upstream_response_time: time it took for upstream (MyService) to handle request.
  # - upstream_cache_status: status of accessing a response cache.
  #   The status can be either `MISS`, `BYPASS`, `EXPIRED`, `STALE`, `UPDATING`, `REVALIDATED` or `HIT`.
  log_format custom '$remote_addr - $remote_user [$time_local] '
                      '"$request" $status $body_bytes_sent '
                      '"$http_referer" "$http_user_agent" '
                      '$request_time $upstream_response_time $upstream_cache_status';

  # - levels: nestiness of cache directories, ie /cache/c/29/b7f54b2df7773722d382f4809d65029c
  # - keys_zone: named memory structure to hold keys, 1 MB allows to store around 8 thousand keys.
  # - max_size: max size of cache entries on disk.
  # — inactive: amount of time a key/value pair can live in cache (even if it was accessed recently).
  # - use_temp_path: on/off to use nginx configured tmp directory.
  proxy_cache_path /opt/nginx-cache/ levels=1:2 keys_zone=myservice_cache:10m max_size=10g inactive=24h use_temp_path=off;

  server {
    listen 80;
    server_name "";

    access_log /var/log/nginx/access.log custom;

    # Caching setup is quite aggressive, we assume MyService to be append-only.
    proxy_cache myservice_cache;

    # Cache authenticated requests by making basic auth `Authorization` header part of a cache key.
    proxy_cache_key "$http_authorization$request_uri";

    proxy_cache_methods GET HEAD;

    # Cache `200` responses for 24 hours.
    proxy_cache_valid 200 24h;

    # Cache `404` responses for 10 minutes, 404 might be resolved to 200 later.
    proxy_cache_valid 404 10m;

    # Allow parallel cache update while we serve stale cache entry.
    proxy_cache_background_update on;

    # Return cache if upstream is down or Nginx is updating the cache atm.
    proxy_cache_use_stale timeout updating http_500 http_502 http_503 http_504;

    proxy_read_timeout 10s;
    proxy_connect_timeout 1s;

    location / {
      proxy_pass http://myservice;
    }

    location /nginx-health {
      access_log off;
      return 200;
    }
  }
}

It runs as Kubernetes pods:

containers:
- name: nginx
  image: docker.io/nginx:1.15.8
  volumeMounts:
  - name: nginx-conf
    mountPath: /etc/nginx/
  - name: nginx-cache
    mountPath: /opt/nginx-cache
  ports:
  - containerPort: 80
    name: http
  resources:
    requests:
      memory: "1G"
      cpu: "1"
    limits:
      memory: "1G"
      cpu: "1"
  livenessProbe:
    httpGet:
      path: "/nginx-health"
      port: 80
    initialDelaySeconds: 1
    periodSeconds: 3
    failureThreshold: 1
    timeoutSeconds: 1
  readinessProbe:
    httpGet:
      path: "/nginx-health"
      port: 80
    initialDelaySeconds: 2
    periodSeconds: 1
    failureThreshold: 1
    timeoutSeconds: 1

Observations

We observe 3 problems with this setup.

Problem 1: Nginx container terminates itself sometimes

We see Nginx container terminating itself with code 0 with no warnings or errors in logs.

State:          Running
      Started:      Thu, 24 Jan 2019 11:26:18 -0800
    Last State:     Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Thu, 24 Jan 2019 11:00:25 -0800
      Finished:     Thu, 24 Jan 2019 11:26:18 -0800
    Ready:          True
    Restart Count:  1

Logs just have some regular requests it serves all the time.

Problem 2: Nginx times out on a static return 200 health check sometimes

We see Nginx container timing out (1 second) very very simple health check:

location /nginx-health {
  access_log off;
  return 200;
}

We used to log it, but nothing stood out when we saw time outs.

Problem 3: Nginx containers consume ~96% of given memory all the time

We're giving container 1GB of memory as limit.
For some reason Nginx consumes pretty much all of it on production load which is not anything special — 60 RPS or so usually and keeps consuming it in after hours when we don't get any requests.

Moreover, we can't figure out what consumes memory in the container:

I've blamed shared memory first, but it seems to be fine too:

root@caching-proxy-0:/# free -m
              total        used        free      shared  buff/cache   available
Mem:          70339       10145       21278          10       38915       59570
Swap:             0           0           0

Exactly 10mb I set for cache key_zone…

Expectations

1. Nginx container shouldn't terminate itself with exit code 0
2. Nginx container shouldn't terminate itself without any useful logs
3. Nginx container shouldn't consume 1GB of memory in this setup (or I guess I just expect it to be within 10-200MB usually)

---

We're happy to provide additional information.

And we're really curious on finding out where is this memory consumption coming from, the only thing left on mind is Linux page cache…

I've spent hours reading everything I could find, these links seem to be the only ones close to what we observe:

• https://github.com/moby/moby/issues/21759
• https://stackoverflow.com/questions/41048742/nginx-content-caching-causing-docker-memory-spike

But still I can't figure how to fix it.

All problems seem to be caused by Problem №3: memory consumption of the container

Start with the nginx configuration and Kubernetes container manifest shown in the report, then inspect the nginx:1.15.8 image and the linked Moby issue. Reproduce the cache workload while measuring container memory, process state, and probe failures. Done means explaining all three symptoms and identifying a verified configuration or image-level fix.