openresty / openresty/openresty

One random worker consumes 100% cpu

Open
#912 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C
Stars
14k
Forks
1.6k
PR merge metrics
No merged PRs in 30d

Description

Hello
One random worker of openresty occasionally consumes 100% of the cpu, openresty version is 1.19.3。Use sample-bt tool to catch nginx c backtraces, From the flame graph we can see that ngx_ssl_shutdown take too much cpu time, we also find some errors in error log。 How can we debug this?
we use intel QAT with async mode nginx, detail info is from https://github.com/intel/asynch_mode_nginx,nginx config file related to ssl is as follows:

nginx config

load_module modules/ngx_ssl_engine_qat_module.so;
lua_ssl_verify_depth 5;
ssl_session_timeout 86400;
proxy_ssl_name $upstream_host;
proxy_ssl_server_name on;
ssl_asynch on;
ssl_certificate cert/default.pem;
ssl_certificate_key cert/default.key;
ssl_session_cache shared:SSL:20m;
ssl_session_timeout 10m;
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers TLS_AES_128_GCM_SHA256:ECDHE-RSA-AES128-GCM-SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:HIGH:!ADH:!EXPORT56:!RC4:!NULL:!aNULL:!DH:!DHE:+MEDIUM;
ssl_prefer_server_ciphers on;
ssl_session_tickets off;

error log

023/06/28 09:33:20 [crit] 53540#53540: *83943154523 SSL_do_handshake() failed (SSL: error:141B306E:SSL routines:tls_collect_extensions:bad extension) while SSL handshaking,
2023/06/28 09:33:20 [crit] 53517#53517: *83943181752 SSL_do_handshake() failed (SSL: error:141B306E:SSL routines:tls_collect_extensions:bad extension) while SSL handshaking,
2023/06/28 09:33:23 [crit] 53480#53480: *83943257007 SSL_do_handshake() failed (SSL: error:800640A8:lib(128):aes_gcm_tls_cipher:gcm tag verify failure error:1408F119:SS routines:ssl3_get_record:decryption failed or bad record mac) while SSL handshaking,
2023/06/28 09:33:23 [crit] 53487#53487: *83943260352 SSL_read() failed (SSL: error:800640A8:lib(128):aes_gcm_tls_cipher:gcm tag verify failure error:1408F119:SSL routines:ssl3_get_record:decryption failed or bad record mac) while waiting for request,
2023/06/28 09:33:46 [crit] 10802#10802: *83944140841 SSL_read() failed (SSL: error:800640A8:lib(128):aes_gcm_tls_cipher:gcm tag verify failure error:1408F119:SSL routines:ssl3_get_record:decryption failed or bad record mac) while keepalive
2023/06/28 09:33:57 [crit] 53509#53509: *83944638838 SSL_do_handshake() failed (SSL: error:141B306E:SSL routines:tls_collect_extensions:bad extension) while SSL handshaking
2023/06/28 09:34:03 [crit] 53570#53570: *83944840894 SSL_do_handshake() failed (SSL: error:141CF0D1:SSL routines:tls_parse_ctos_key_share:missing supported groups extension) while SSL handshaking

flame

image

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No source file or test is identified. Start with the sample-bt output and flame graph around ngx_ssl_shutdown, then correlate it with the SSL error-log entries while ssl_asynch and the Intel QAT module are enabled. Done means a reproducible diagnosis of the high-CPU worker and documented evidence for the cause.

Written by the indexing model from the issue text.

Assessment

Tech stack
c
Domain
networking, performance, security
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.