openresty / openresty/stream-lua-nginx-module

worker process suffers CPU 100% usage

Open
#167 3 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C
Stars
749
Forks
210
Avg merge
4h 31m
Merged PRs (30d)
3

Description

Hello!

The worker process CPU usage could be 100% if we use the following configuration pattern:

stream {
    server {
        listen 3105 reuseport;

        lua_check_client_abort on; # must be enabled

        content_by_lua_block {
            local sock, err = ngx.req.socket(true)

            sock:settimeout(1800 * 1000)

            local readline = sock:receiveuntil("\n")

            while not ngx.worker.exiting() do
                local line, err = readline(1)
                ngx.log(ngx.ERR, "line: ", line)
                if not line then
                    if err then
                        ngx.log(ngx.ERR, err)
                        break
                    end
                else
                    ngx.sleep(5000) -- other yieldable API is also applicable.
                end
            end
        }
    }
 }

Building a connection through nc and sending some characters then closing it, in the meanwhile, observing the CPU usage by top.

$ nc 127.0.0.1 3105
asjdkl
zxckj
^C

Now you can find the CPU usage reaches 100%.

It's irrelevant with the version of ngx_stream_lua, you can reproduce this problem even with the newest OpenResty.

After investigating the internal of stream-lua-nginx-module, I found it was caused by the indefinitely trigger of ngx_stram_lua_check_broken_connection, which doesn't the check ev->pending_eof flag.

BTW, this issue was found by @tianchaijz.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at ngx_stram_lua_check_broken_connection and trace the worker event handling involved in the provided stream configuration. Reproduce the issue with the nginx configuration and nc commands, then verify that closing the connection no longer causes 100% worker CPU usage.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, nginx
Domain
backend, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.