openresty / openresty/lua-nginx-module

Memory overhead when using lua-nginx-module

Open
#872 17 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C
Stars
11.8k
Forks
2.1k
Avg merge
6h 1m
Merged PRs (30d)
6

Description

I have nginx with lua-nginx-module.
When I send range requests to nginx with lua-nginx-module and enabled http_slice_module, nginx takes all free memory and does not free even if all connections closed.
Without lua-nginx-module nginx does not use so much memory, one of worker processes can use some memory and free it during work.

$ nginx -V
nginx version: nginx/1.11.2
built by gcc 4.8.4 (Ubuntu 4.8.4-2ubuntu1~14.04.3)
built with OpenSSL 1.0.1f 6 Jan 2014
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-http_ssl_module --with-http_realip_module --with-http_addition_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_random_index_module --with-http_secure_link_module --with-http_stub_status_module --with-http_auth_request_module --with-http_xslt_module=dynamic --with-http_image_filter_module=dynamic --with-http_geoip_module=dynamic --with-http_perl_module=dynamic --add-dynamic-module=debian/extra/njs-ef2b708510b1/nginx --with-threads --with-stream --with-stream_ssl_module --with-http_slice_module --with-mail --with-mail_ssl_module --with-file-aio --with-ipv6 --with-http_v2_module --with-cc-opt='-g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security' --with-ld-opt='-Wl,-Bsymbolic-functions -Wl,-z,relro' --add-module=ngx_devel_kit-0.3.0 --add-module=lua-nginx-module-0.10.6

libluajit-5.1-2 version:

$ ldd /usr/sbin/nginx | grep lua
    libluajit-5.1.so.2 => /usr/lib/x86_64-linux-gnu/libluajit-5.1.so.2 
$ ls -al /usr/lib/x86_64-linux-gnu/libluajit-5.1.so.2
lrwxrwxrwx 1 root root 22 Jun  9  2013 /usr/lib/x86_64-linux-gnu/libluajit-5.1.so.2 -> libluajit-5.1.so.2.0.2

With libluajit-5.1.so.2.0.4 memory usage same as with libluajit-5.1.so.2.0.2.

For example nginx with lua-nginx-module:

memory usage before test:

$ ps -e -orss=,args= | grep nginx: | awk 'BEGIN{sum=0;}{sum+=$1; print $0; }END{print "Total memory = ", sum;}'
 2712 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
13340 nginx: worker process
13340 nginx: worker process
13340 nginx: worker process
13340 nginx: worker process
13340 nginx: worker process
13340 nginx: worker process
13340 nginx: worker process
13340 nginx: worker process
13340 nginx: worker process
13340 nginx: worker process
13340 nginx: worker process
13340 nginx: worker process
13340 nginx: worker process
13340 nginx: worker process
13340 nginx: worker process
13340 nginx: worker process
 3360 nginx: cache manager process
 2948 nginx: cache loader process
  944 grep --color=auto nginx:
Total memory =  223404

memory usage after test (all requests finished and all connections to nginx closed):

$ ps -e -orss=,args= | grep nginx: | awk 'BEGIN{sum=0;}{sum+=$1; print $0; }END{print "Total memory = ", sum;}'
 2372 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
2033368 nginx: worker process
1993736 nginx: worker process
2010688 nginx: worker process
2072408 nginx: worker process
1899844 nginx: worker process
2036096 nginx: worker process
2081336 nginx: worker process
1829648 nginx: worker process
1992792 nginx: worker process
1705992 nginx: worker process
2074204 nginx: worker process
1805180 nginx: worker process
2003932 nginx: worker process
1902636 nginx: worker process
2081448 nginx: worker process
2048644 nginx: worker process
 3008 nginx: cache manager process
  804 grep --color=auto nginx:
Total memory =  31578136

nginx config:

worker_processes  16;
worker_rlimit_nofile 1000000;

events {
}

http {
    output_buffers            1 1m;

    log_format main '$remote_addr [$time_local] "$request" $status $bytes_sent'
                    ' "$request_time" "$upstream_cache_status" "$http_range"';

    proxy_cache_path       /var/cache/nginx/proxy_cache levels=2:2 keys_zone=proxy_slice:500m inactive=3d use_temp_path=off;

    server {
      listen      127.0.0.1:8080 default_server backlog=65535;
      access_log  /var/log/nginx/proxy_slice.log main;

      slice             1m;
      proxy_cache       proxy_slice;
      proxy_cache_key   "$request_uri|$slice_range";
      proxy_cache_valid 200 206 1d;
      proxy_cache_valid any 0;
      proxy_set_header  Range    $slice_range;
      proxy_ignore_client_abort on;
      proxy_http_version 1.1;

      location / {
        proxy_pass http://127.0.0.1:8088;
      }
    }

    server {
      listen 127.0.0.1:8088;

      access_log /var/log/nginx/storage.log main;
      rewrite ^.*$ /1.data last;

      location / {
        root /var/data;
      }
    }
}

How to reproduce:

$ mkdir /var/data; 
$ dd if=/dev/urandom of=/var/data/1.data bs=1M count=4500

Command for test:

$ while true; do curl -H "Range: bytes=$(echo $(od -An -N3 -i /dev/urandom)-$(( 100 * $(od -An -N3 -i /dev/urandom) )))" "http://127.0.0.1:8080/$(head /dev/urandom | tr -dc A-Za-z0-9 | head -c 13)" > /dev/null ; done

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the reported RSS growth with the supplied nginx configuration, http_slice_module, lua-nginx-module, and range-request loop, then compare behavior with lua-nginx-module disabled. Trace the interaction between the two modules and verify memory after connections close; done means identifying the retention cause and demonstrating that the corrected behavior persists under the reproduction.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, lua, nginx
Domain
backend, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.