openresty / openresty/openresty
One random worker consumes 100% cpu
Nobody has claimed this yet.
- Dominant language
- C
- Stars
- 14k
- Forks
- 1.6k
- PR merge metrics
- No merged PRs in 30d
Description
Hello
One random worker of openresty occasionally consumes 100% of the cpu, openresty version is 1.19.3。Use sample-bt tool to catch nginx c backtraces, From the flame graph we can see that ngx_ssl_shutdown take too much cpu time, we also find some errors in error log。 How can we debug this?
we use intel QAT with async mode nginx, detail info is from https://github.com/intel/asynch_mode_nginx,nginx config file related to ssl is as follows:
nginx config
load_module modules/ngx_ssl_engine_qat_module.so;
lua_ssl_verify_depth 5;
ssl_session_timeout 86400;
proxy_ssl_name $upstream_host;
proxy_ssl_server_name on;
ssl_asynch on;
ssl_certificate cert/default.pem;
ssl_certificate_key cert/default.key;
ssl_session_cache shared:SSL:20m;
ssl_session_timeout 10m;
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers TLS_AES_128_GCM_SHA256:ECDHE-RSA-AES128-GCM-SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:HIGH:!ADH:!EXPORT56:!RC4:!NULL:!aNULL:!DH:!DHE:+MEDIUM;
ssl_prefer_server_ciphers on;
ssl_session_tickets off;
error log
023/06/28 09:33:20 [crit] 53540#53540: *83943154523 SSL_do_handshake() failed (SSL: error:141B306E:SSL routines:tls_collect_extensions:bad extension) while SSL handshaking,
2023/06/28 09:33:20 [crit] 53517#53517: *83943181752 SSL_do_handshake() failed (SSL: error:141B306E:SSL routines:tls_collect_extensions:bad extension) while SSL handshaking,
2023/06/28 09:33:23 [crit] 53480#53480: *83943257007 SSL_do_handshake() failed (SSL: error:800640A8:lib(128):aes_gcm_tls_cipher:gcm tag verify failure error:1408F119:SS routines:ssl3_get_record:decryption failed or bad record mac) while SSL handshaking,
2023/06/28 09:33:23 [crit] 53487#53487: *83943260352 SSL_read() failed (SSL: error:800640A8:lib(128):aes_gcm_tls_cipher:gcm tag verify failure error:1408F119:SSL routines:ssl3_get_record:decryption failed or bad record mac) while waiting for request,
2023/06/28 09:33:46 [crit] 10802#10802: *83944140841 SSL_read() failed (SSL: error:800640A8:lib(128):aes_gcm_tls_cipher:gcm tag verify failure error:1408F119:SSL routines:ssl3_get_record:decryption failed or bad record mac) while keepalive
2023/06/28 09:33:57 [crit] 53509#53509: *83944638838 SSL_do_handshake() failed (SSL: error:141B306E:SSL routines:tls_collect_extensions:bad extension) while SSL handshaking
2023/06/28 09:34:03 [crit] 53570#53570: *83944840894 SSL_do_handshake() failed (SSL: error:141CF0D1:SSL routines:tls_parse_ctos_key_share:missing supported groups extension) while SSL handshaking
flame
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No source file or test is identified. Start with the sample-bt output and flame graph around ngx_ssl_shutdown, then correlate it with the SSL error-log entries while ssl_asynch and the Intel QAT module are enabled. Done means a reproducible diagnosis of the high-CPU worker and documented evidence for the cause.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c
- Domain
- networking, performance, security
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100