openresty / openresty/luajit2

Nginx process endlessly stuck in err_unwind function causes 100% CPU

Open
#97 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C
Stars
1.4k
Forks
252
Avg merge
7d 11h
Merged PRs (30d)
3

Description

openresty version : openresty-1.13.6.1
error.log:nginx: lua atpanic: Lua VM crashed, reason: not enough memory
STACK1:
Traceback (most recent call last):

File "luajit21.py", line 588, in invoke

bt = lj_debug_dumpstack(L, 0, 30, base, full)

File "luajit21.py", line 423, in lj_debug_dumpstack

frame, size = lj_debug_frame(L, base, level, bot)

File "luajit21.py", line 218, in lj_debug_frame

if frame_gc(frame) == obj2gco(L):

File "luajit21.py", line 177, in frame_gc

return gcref(frame['fr']['func'])

File "luajit21.py", line 170, in gcref

return r['gcptr32'].cast(typ("uintptr_t")).cast(typ("GCobj*"))

g
gdb.MemoryError: Cannot access memory at address 0xfffffffffffffff8
E
Error occurred in Python command: Cannot access memory at address 0xfffffffffffffff8

STACK2:
#0 err_unwind (L=L@entry=0x41e9f378, stopcf=0x7ffcddad1520, errcode=errcode@entry=0) at lj_err.c:109

#1 0x00007f988b3504a5 in lj_err_unwind_dwarf (version=, actions=1, uexclass=5500374307216568836, uex=0x7f988c51e760, ctx=0x7ffcddad1170) at lj_err.c:240

#2 0x00007f988a347c33 in _Unwind_RaiseException () from /lib64/libgcc_s.so.1

#3 0x00007f988b350329 in err_raise_ext (errcode=4) at lj_err.c:301

#4 lj_err_throw (L=L@entry=0x41e9f378, errcode=errcode@entry=4) at lj_err.c:515

#5 0x00007f988b350563 in lj_err_mem (L=L@entry=0x41e9f378) at lj_err.c:551

#6 0x00007f988b350000 in lj_mem_newgco (L=L@entry=0x41e9f378, size=20) at lj_gc.c:833

#7 0x00007f988b3530ec in func_newL (L=L@entry=0x41e9f378, pt=pt@entry=0x41ee8010, env=0x41ea09d8) at lj_func.c:122

#8 0x00007f988b353310 in lj_func_newL_gc (L=0x41e9f378, pt=, parent=0x415cde28) at lj_func.c:160

#9 0x00007f988b34bbe9 in lj_BC_FNEW () from /usr/local/NSP/luajit/lib/libluajit-5.1.so.2

#10 0x00007f988b35d0bd in lua_pcall (L=L@entry=0x41e9f378, nargs=nargs@entry=0, nresults=nresults@entry=1, errfunc=errfunc@entry=0) at lj_api.c:1129

#11 0x00000000004f9c75 in ngx_http_lua_cache_load_code (log=log@entry=0x2062db70, L=L@entry=0x41e9f378, key=key@entry=0x21327480 "nhlf_cdbb7e422b70672672ea5d04e77ae42c")

at ../ngx_lua-0.10.11/src/ngx_http_lua_cache.c:56

#12 0x00000000004f9f1c in ngx_http_lua_cache_loadfile (log=0x2062db70, L=L@entry=0x41e9f378, script=0x31727c38 "/usr/local/NSP/etc/router/router_ext.luac",

cache_key=0x21327480 "nhlf_cdbb7e422b70672672ea5d04e77ae42c") at ../ngx_lua-0.10.11/src/ngx_http_lua_cache.c:232

#13 0x00000000004fb9c7 in ngx_http_lua_access_handler_file (r=0x31726cf0) at ../ngx_lua-0.10.11/src/ngx_http_lua_accessby.c:219

#14 0x00000000004fb5d0 in ngx_http_lua_access_handler (r=0x31726cf0) at ../ngx_lua-0.10.11/src/ngx_http_lua_accessby.c:163

#15 0x0000000000459dbb in ngx_http_core_access_phase (r=, ph=0x17ab770) at src/http/ngx_http_core_module.c:1087

#16 0x000000000045562d in ngx_http_core_run_phases (r=r@entry=0x31726cf0) at src/http/ngx_http_core_module.c:862

#17 0x0000000000455722 in ngx_http_handler (r=r@entry=0x31726cf0) at src/http/ngx_http_core_module.c:845

#18 0x000000000046024e in ngx_http_process_request (r=0x31726cf0) at src/http/ngx_http_request.c:1986

#19 0x0000000000460ac7 in ngx_http_process_request_line (rev=0x7f981e7851c8) at src/http/ngx_http_request.c:1076

#20 0x000000000044a09c in ngx_epoll_process_events (cycle=, timer=, flags=) at src/event/modules/ngx_epoll_module.c:968

#21 0x00000000004413df in ngx_process_events_and_timers (cycle=cycle@entry=0x15ac050) at src/event/ngx_event.c:266

#22 0x0000000000447f21 in ngx_worker_process_cycle (cycle=cycle@entry=0x15ac050, data=data@entry=0x17) at src/os/unix/ngx_process_cycle.c:821

#23 0x00000000004468c6 in ngx_spawn_process (cycle=cycle@entry=0x15ac050, proc=proc@entry=0x447ea0 <ngx_worker_process_cycle>, data=data@entry=0x17,

name=name@entry=0x5a58e5 "worker process", respawn=respawn@entry=-4) at src/os/unix/ngx_process.c:198

#24 0x0000000000448240 in ngx_start_worker_processes (cycle=cycle@entry=0x15ac050, n=32, type=type@entry=-4) at src/os/unix/ngx_process_cycle.c:396

#25 0x0000000000448ef5 in ngx_master_process_cycle (cycle=0x15ac050, cycle@entry=0x15a6230) at src/os/unix/ngx_process_cycle.c:250

#26 0x0000000000421583 in main (argc=, argv=) at src/core/nginx.c:387

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with err_unwind in lj_err.c and trace the reported lj_err_unwind_dwarf, err_raise_ext, lj_err_mem, and lj_mem_newgco frames through ngx_http_lua_cache_load_code in ngx_http_lua_cache.c. Use the supplied stack and Lua VM out-of-memory message to investigate why unwinding loops and leaves a worker at 100% CPU; done means the reported endless loop is explained and no longer occurs.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, lua
Domain
backend, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.