openresty / openresty/lua-nginx-module

Lua script freeze nginx worker

Open
#2,441 4 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C
Stars
11.8k
Forks
2.1k
Avg merge
6h 1m
Merged PRs (30d)
6

Description

Hello! I have a trouble with reload nginx-ingress.
I have a theory that k8s has so many ingress-crd inside, that lua blocking nginx worker and clients see 502 error.
I tried to re-create this scenario and it really possible to make this on test stand

pid /tmp/nginx/nginx.pid;
daemon off;
worker_processes 2;
worker_shutdown_timeout 240s ;
events {
	multi_accept        on;
	worker_connections  16384;
	use                 epoll;
}
http {
    lua_shared_dict dummy 1m;
    server {
        listen 8080;
        location / {
            return 200 "OK\n";
        }
    }
    init_worker_by_lua_block {
        ngx.log(ngx.ERR, "Heavy math start")
        local acc = 0
        for i = 1, 1e8 do
            acc = acc + math.sqrt(i) * math.sin(i) / (i + 1)
        end
        ngx.log(ngx.ERR, "Heavy math done")
    }
}
docker run --rm -it -v $(pwd)/nginx.conf:/etc/nginx/nginx.conf:ro -p 80:8080 registry.k8s.io/ingress-nginx/controller:v1.13.0 nginx

And you will see that depends on power of your machine nginx becomes ready after 10-20seconds.

Same situations happens when you will make nginx -s reload, master proccess started to route traffic to new workers BEFORE they becomes ready for it.

May be there is any possible solution to force nginx master process wait for complete initialization init_worker_by_lua_block { } and only after that started to route traffic on the new workers?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No source file or test is named. Start by reproducing the supplied nginx configuration and reload sequence, then trace init_worker_by_lua_block and nginx worker startup behavior. Done would require an agreed solution that prevents traffic from reaching workers before initialization completes, with regression coverage for reloads.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, lua, nginx
Domain
backend, infrastructure
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.