influxdata / influxdata/influxdb

InfluxDB OSS 2.7.12 Deadlock randomly

Open
#26,817 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

I have 5 OSS 2.7.12 InfluxDBs running in my distributed environment, they all dead randomly in two weeks.
The first dead InfluxDB only running one day.
CPU is 0, memory is not moved, all api cannot get responsed, even the ping api.
The phenomenon of this hang is very similar to deadlock.
Restart the Influxd.exe, it will return to work.

__Response with InfluxCli:__
Type comand (influx.exe org list) with InfluxCli with below response:
Error: failed to list orgs: Get "http://localhost:8086/api/v2/orgs: dial tcp [::1]:8086: connectex: No connection could be made because the target machine actively refused it.
See the netstat -ano, the 8086 is binding with the InfluxD pid. State is fine.

__Dump1:__
Catch dump file of one application which has been running 10 days, analyze.
!runaway, cannot see any thead is down
User Mode Time
Thread Time
27:2894 0 days 1:13:13.000
28:2898 0 days 1:12:41.390
15:23b8 0 days 1:12:37.593
16:23bc 0 days 1:10:36.343
22:23d8 0 days 1:09:51.265
14:23b4 0 days 1:08:21.156
7:2390 0 days 1:07:37.890
30:28b8 0 days 1:05:44.453
34:28c8 0 days 1:05:44.312
2:219c 0 days 1:05:17.000
19:23cc 0 days 1:05:05.578
13:23b0 0 days 1:05:05.500
25:288c 0 days 1:04:58.890
12:23ac 0 days 1:04:32.609
3:21a0 0 days 1:04:19.375
20:23d0 0 days 1:03:39.625
17:23c4 0 days 1:03:36.984
32:28c0 0 days 1:01:45.062
6:238c 0 days 1:00:28.578
35:28d0 0 days 1:00:23.625
26:2890 0 days 0:56:45.343
29:28b4 0 days 0:55:59.375
9:239c 0 days 0:55:22.937
11:23a8 0 days 0:55:08.921
33:28c4 0 days 0:54:47.609
24:2830 0 days 0:54:20.109
21:23d4 0 days 0:54:07.343
31:28bc 0 days 0:52:59.828
23:282c 0 days 0:52:04.593
10:23a0 0 days 0:51:49.109
36:28cc 0 days 0:50:48.437
18:23c8 0 days 0:49:49.687
8:2398 0 days 0:41:22.578
4:21a4 0 days 0:24:57.531
37:28d4 0 days 0:11:20.171
1:2184 0 days 0:01:32.546
0:199c 0 days 0:00:00.062
5:21c4 0 days 0:00:00.000

~*kb, I can see thread 23 is not suspend, I don't have symbol, cannot know what it was doing.
0:0> ~*kb

0 Id: 1998.199c Suspend: 0 Teb: 00000000`002e7000 Unfrozen
RetAddr : Args to Child : Call Site
00 00007ffe`008bda3e : 00000000`00410f96 000000c0`0006c008 00000000`06ff03c0 000000c0`0006c008 : ntdll!ZwWaitForSingleObject+0x14
01 00000000`004821c9 : 00000000`06ff06a0 000000c0`00a07500 00000000`00000000 00000000`00000164 : KERNELBASE!WaitForSingleObjectEx+0x8e
02 00000000`0048054d : 00000000`00000000 00000000`0733fd80 00000000`0044be79 000000c0`0007c508 : influxd+0x821c9
03 00000000`004805c3 : 00000000`000001a8 000000c0`00a07500 00000000`0733fda8 00000000`00485885 : influxd+0x8054d
......

1 Id: 1998.2184 Suspend: 0 Teb: 00000000`002ef000 Unfrozen
RetAddr : Args to Child : Call Site
00 00007ffe`008bda3e : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!ZwWaitForSingleObject+0x14
01 00000000`004821c9 : 000000c0`000beae8 00000000`002ef000 00000000`00000000 00000000`000003f8 : KERNELBASE!WaitForSingleObjectEx+0x8e
02 00000000`0048054d : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : influxd+0x821c9
03 00000000`00480601 : 00000000`4d2dfb98 00000000`00000000 00000000`4d2dfbc0 00000000`4d2dfbd0 : influxd+0x8054d
......

2 Id: 1998.219c Suspend: 0 Teb: 00000000`002f1000 Unfrozen
RetAddr : Args to Child : Call Site
00 00007ffe`008bda3e : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!ZwWaitForSingleObject+0x14
01 00000000`004821c9 : 000000c0`000bf2e8 00000000`00000000 00000000`00000000 00000000`00000194 : KERNELBASE!WaitForSingleObjectEx+0x8e
02 00000000`0048054d : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : influxd+0x821c9
03 00000000`00480601 : 00000000`4d4dfb28 00000000`00000000 00000000`00000000 00000000`4d4dfb60 : influxd+0x8054d
......

3 Id: 1998.21a0 Suspend: 0 Teb: 00000000`002f3000 Unfrozen
RetAddr : Args to Child : Call Site
00 00007ffe`008bda3e : 00000000`4d6dfd40 00000000`0043d825 00000000`02907f90 00000000`000009ac : ntdll!ZwWaitForSingleObject+0x14
01 00000000`004821c9 : 000000c0`000bfae8 00000000`002f3000 00000000`00000000 00000000`00000188 : KERNELBASE!WaitForSingleObjectEx+0x8e
02 00000000`0048054d : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : influxd+0x821c9
03 00000000`00480601 : 00000000`4d6dfa30 00000000`00000000 00000000`4d6dfa68 00000000`0043c91b : influxd+0x8054d
......

4 Id: 1998.21a4 Suspend: 0 Teb: 00000000`002f5000 Unfrozen
RetAddr : Args to Child : Call Site
00 00007ffe`008bda3e : 00000000`4d8dfd40 00000000`0043d825 00000000`02907f90 00000000`000008d0 : ntdll!ZwWaitForSingleObject+0x14
01 00000000`004821c9 : 000000c0`001002e8 00000000`002f5000 00000000`00000000 00000000`000001ac : KERNELBASE!WaitForSingleObjectEx+0x8e
02 00000000`0048054d : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : influxd+0x821c9
03 00000000`00480601 : 00000000`4d8dfa30 00000000`00000000 00000000`4d8dfa68 00000000`0043c91b : influxd+0x8054d
......

5 Id: 1998.21c4 Suspend: 0 Teb: 00000000`002f7000 Unfrozen
RetAddr : Args to Child : Call Site
00 00007ffe`008bda3e : 00000000`00100003 00007ffe`008d7d04 00000000`00000000 00000000`00000000 : ntdll!ZwWaitForSingleObject+0x14
01 00000000`004821c9 : 000000c0`000c62e8 000000c0`000b2c40 00000000`00000000 00000000`000001cc : KERNELBASE!WaitForSingleObjectEx+0x8e
02 00000000`0048054d : 000000c0`000c62e8 00000000`4dadfce0 00000000`4dadfd20 00000000`0048054d : influxd+0x821c9
03 00000000`00480601 : 00000000`4dadfd20 00000000`00000000 00000000`4dadfd58 00000000`0043c91b : influxd+0x8054d
......

6 Id: 1998.238c Suspend: 0 Teb: 00000000`002f9000 Unfrozen
RetAddr : Args to Child : Call Site
00 00007ffe`008bda3e : 00000000`4dd1fd40 00000000`0043d825 00000000`02907f90 00000000`00000d0c : ntdll!ZwWaitForSingleObject+0x14
01 00000000`004821c9 : 000000c0`004902e8 00000000`002f9000 00000000`00000000 00000000`0000038c : KERNELBASE!WaitForSingleObjectEx+0x8e
02 00000000`0048054d : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : influxd+0x821c9
03 00000000`00480601 : 00000000`4dd1fa30 00000000`00000000 00000000`4dd1fa68 00000000`0043c91b : influxd+0x8054d
......

7 Id: 1998.2390 Suspend: 0 Teb: 00000000`002fb000 Unfrozen
RetAddr : Args to Child : Call Site
00 00007ffe`008bda3e : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!ZwWaitForSingleObject+0x14
01 00000000`004821c9 : 000000c0`005002e8 00000000`002fb000 00000000`00000000 00000000`00000394 : KERNELBASE!WaitForSingleObjectEx+0x8e
02 00000000`0048054d : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : influxd+0x821c9
03 00000000`00480601 : 00000000`4df5fa30 00000000`00000000 00000000`4df5fa68 00000000`0043c91b : influxd+0x8054d
......

8 Id: 1998.2398 Suspend: 0 Teb: 00000000`002fd000 Unfrozen
RetAddr : Args to Child : Call Site
00 00007ffe`008bda3e : 00000000`00000000 00000000`00000000 00000000`4e15f900 00000000`00480601 : ntdll!ZwWaitForSingleObject+0x14
01 00000000`004821c9 : 000000c0`005802e8 00000000`0046305b 00000000`00000000 00000000`0000039c : KERNELBASE!WaitForSingleObjectEx+0x8e
02 00000000`0048054d : 00000000`004282a5 00000000`060b47a0 00000000`4e15fb18 00000000`4e15fb50 : influxd+0x821c9
03 00000000`00480601 : 00000000`4e15fb28 00000000`00000000 00000000`4e15fb50 00000000`4e15fb60 : influxd+0x8054d
......

9 Id: 1998.239c Suspend: 0 Teb: 00000000`002ff000 Unfrozen
RetAddr : Args to Child : Call Site
00 00007ffe`008bda3e : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!ZwWaitForSingleObject+0x14
01 00000000`004821c9 : 000000c0`006002e8 00000000`00000000 00000000`00000000 00000000`00000384 : KERNELBASE!WaitForSingleObjectEx+0x8e
02 00000000`0048054d : 00000000`00000000 00000000`00000000 00000000`4e39fb20 00000000`00410f96 : influxd+0x821c9
03 00000000`00480601 : 00000000`4e39fb28 00000000`00000000 00000000`4e39fb68 00000000`4e39fb60 : influxd+0x8054d
......

10 Id: 1998.23a0 Suspend: 0 Teb: 00000000`00301000 Unfrozen
RetAddr : Args to Child : Call Site
00 00007ffe`008bda3e : 00000000`4e59fd40 00000000`0043d825 00000000`02907f90 00000000`00000d0c : ntdll!ZwWaitForSingleObject+0x14
01 00000000`004821c9 : 000000c0`006802e8 00000000`00301000 00000000`00000000 00000000`00000378 : KERNELBASE!WaitForSingleObjectEx+0x8e
02 00000000`0048054d : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : influxd+0x821c9
03 00000000`00480601 : 00000000`4e59fa30 00000000`00000000 00000000`4e59fa68 00000000`0043c91b : influxd+0x8054d
......

11 Id: 1998.23a8 Suspend: 0 Teb: 00000000`00303000 Unfrozen
RetAddr : Args to Child : Call Site
00 00007ffe`008bda3e : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!ZwWaitForSingleObject+0x14
01 00000000`004821c9 : 000000c0`007002e8 00000000`00303000 00000000`00000000 00000000`0000034c : KERNELBASE!WaitForSingleObjectEx+0x8e
02 00000000`0048054d : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : influxd+0x821c9
03 00000000`00480601 : 00000000`4e7dfa30 00000000`00000000 00000000`4e7dfa68 00000000`0043c91b : influxd+0x8054d
......

12 Id: 1998.23ac Suspend: 0 Teb: 00000000`00305000 Unfrozen
RetAddr : Args to Child : Call Site
00 00007ffe`008bda3e : 005970d2`00000000 00000000`4ea1fc88 00000000`00449a05 000000c0`00088008 : ntdll!ZwWaitForSingleObject+0x14
01 00000000`004821c9 : 000000c0`007802e8 00000000`00000000 00000000`00000000 00000000`00000370 : KERNELBASE!WaitForSingleObjectEx+0x8e
02 00000000`0048054d : 000000c0`00780008 00000000`4ea1fe40 00000000`0043c979 00000000`00000200 : influxd+0x821c9
03 00000000`004805c3 : 00000000`00000388 000000c0`1a7e96c0 00000000`0047e9a9 000000c0`00780060 : influxd+0x8054d
......

13 Id: 1998.23b0 Suspend: 0 Teb: 00000000`00307000 Unfrozen
RetAddr : Args to Child : Call Site
00 00007ffe`008bda3e : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!ZwWaitForSingleObject+0x14
01 00000000`004821c9 : 000000c0`008002e8 00000000`00307000 00000000`00000000 00000000`00000358 : KERNELBASE!WaitForSingleObjectEx+0x8e
02 00000000`0048054d : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : influxd+0x821c9
03 00000000`00480601 : 00000000`4ec5fa30 00000000`00000000 00000000`4ec5fa68 00000000`0043c91b : influxd+0x8054d
......

14 Id: 1998.23b4 Suspend: 0 Teb: 00000000`00309000 Unfrozen
RetAddr : Args to Child : Call Site
00 00007ffe`008bda3e : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!ZwWaitForSingleObject+0x14
01 00000000`004821c9 : 000000c0`008802e8 00000000`00309000 00000000`00000000 00000000`00000368 : KERNELBASE!WaitForSingleObjectEx+0x8e
02 00000000`0048054d : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : influxd+0x821c9
03 00000000`00480601 : 00000000`4ee9f9b8 00000000`00000000 00000000`4ee9f9e0 00000000`4ee9f9f0 : influxd+0x8054d
......

15 Id: 1998.23b8 Suspend: 0 Teb: 00000000`0030b000 Unfrozen
RetAddr : Args to Child : Call Site
00 00007ffe`008bda3e : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!ZwWaitForSingleObject+0x14
01 00000000`004821c9 : 000000c0`009002e8 00000000`0030b000 00000000`00000000 00000000`0000032c : KERNELBASE!WaitForSingleObjectEx+0x8e
02 00000000`0048054d : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : influxd+0x821c9
03 00000000`00480601 : 00000000`4f0dfa30 00000000`00000000 00000000`4f0dfa68 00000000`0043c91b : influxd+0x8054d
......

16 Id: 1998.23bc Suspend: 0 Teb: 00000000`0030d000 Unfrozen
RetAddr : Args to Child : Call Site
00 00007ffe`008bda3e : 0000000f`00000010 00000000`00000000 0002e617`d4c17f94 00000000`00000000 : ntdll!ZwWaitForSingleObject+0x14
01 00000000`004821c9 : 000000c0`009802e8 00000000`4eea2c68 00000000`00000000 00000000`00000360 : KERNELBASE!WaitForSingleObjectEx+0x8e
02 00000000`0048054d : 000000c0`00980008 00000000`4f2dfe40 00000000`0043c979 00000000`00000200 : influxd+0x821c9
03 00000000`004805c3 : 00000000`00000388 000000c0`1a7d8a80 00000000`0047e9a9 000000c0`00980060 : influxd+0x8054d
......

17 Id: 1998.23c4 Suspend: 0 Teb: 00000000`0030f000 Unfrozen
RetAddr : Args to Child : Call Site
00 00007ffe`008bda3e : 00000000`4f51fd40 00000000`0043d825 00000000`02907f90 00000000`0000078c : ntdll!ZwWaitForSingleObject+0x14
01 00000000`004821c9 : 000000c0`00a002e8 00000000`0030f000 00000000`00000000 00000000`00000338 : KERNELBASE!WaitForSingleObjectEx+0x8e
02 00000000`0048054d : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : influxd+0x821c9
03 00000000`00480601 : 00000000`4f51fa30 00000000`00000000 00000000`4f51fa68 00000000`0043c91b : influxd+0x8054d
......

18 Id: 1998.23c8 Suspend: 0 Teb: 00000000`00311000 Unfrozen
RetAddr : Args to Child : Call Site
00 00007ffe`008bda3e : 00000000`4f71fd40 00000000`0043d825 00000000`02907f90 00000000`00000df8 : ntdll!ZwWaitForSingleObject+0x14
01 00000000`004821c9 : 000000c0`00a802e8 00000000`00311000 00000000`00000000 00000000`00000334 : KERNELBASE!WaitForSingleObjectEx+0x8e
02 00000000`0048054d : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : influxd+0x821c9
03 00000000`00480601 : 00000000`4f71fa30 00000000`00000000 00000000`4f71fa68 00000000`0043c91b : influxd+0x8054d
......

19 Id: 1998.23cc Suspend: 0 Teb: 00000000`00313000 Unfrozen
RetAddr : Args to Child : Call Site
00 00007ffe`008bda3e : 00000000`4f95fd40 00000000`0043d825 00000000`02907f90 00000000`00000df8 : ntdll!ZwWaitForSingleObject+0x14
01 00000000`004821c9 : 000000c0`00b002e8 00000000`00313000 00000000`00000000 00000000`00000320 : KERNELBASE!WaitForSingleObjectEx+0x8e
02 00000000`0048054d : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : influxd+0x821c9
03 00000000`00480601 : 00000000`4f95fa30 00000000`00000000 00000000`4f95fa68 00000000`0043c91b : influxd+0x8054d
......

20 Id: 1998.23d0 Suspend: 0 Teb: 00000000`00315000 Unfrozen
RetAddr : Args to Child : Call Site
00 00007ffe`008bda3e : 00000000`0045e199 00000000`060b17a8 00000000`02c8e620 00000000`00693392 : ntdll!ZwWaitForSingleObject+0x14
01 00000000`004821c9 : 000000c0`00b802e8 00000000`4fb9fa68 00000000`00000000 00000000`00000324 : KERNELBASE!WaitForSingleObjectEx+0x8e
02 00000000`0048054d : 0002e617`d45b7eb0 00000000`004282a5 00000000`00000000 0002e617`d45b7eb0 : influxd+0x821c9
03 00000000`00480601 : 00000000`4fb9fb28 00000000`00000000 00000000`0044863e 00000000`4fb9fb60 : influxd+0x8054d
......

21 Id: 1998.23d4 Suspend: 0 Teb: 00000000`00317000 Unfrozen
RetAddr : Args to Child : Call Site
00 00007ffe`008bda3e : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!ZwWaitForSingleObject+0x14
01 00000000`004821c9 : 000000c0`00c002e8 00000000`00317000 00000000`00000000 00000000`00000318 : KERNELBASE!WaitForSingleObjectEx+0x8e
02 00000000`0048054d : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : influxd+0x821c9
03 00000000`00480601 : 00000000`4fddfa30 00000000`00000000 00000000`4fddfa68 00000000`0043c91b : influxd+0x8054d
......

22 Id: 1998.23d8 Suspend: 0 Teb: 00000000`00319000 Unfrozen
RetAddr : Args to Child : Call Site
00 00007ffe`008bda3e : 00000000`5001fd40 00000000`0043d825 00000000`02907f90 00000000`00000e2c : ntdll!ZwWaitForSingleObject+0x14
01 00000000`004821c9 : 000000c0`00c802e8 00000000`00319000 00000000`00000000 00000000`00000348 : KERNELBASE!WaitForSingleObjectEx+0x8e
02 00000000`0048054d : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : influxd+0x821c9
03 00000000`00480601 : 00000000`5001fa30 00000000`00000000 00000000`5001fa68 00000000`0043c91b : influxd+0x8054d
......

**23 Id: 1998.282c Suspend: 0 Teb: 00000000`0031d000 Unfrozen**
RetAddr : Args to Child : Call Site
00 00000000`0043446e : 00000000`00000003 00000000`00000000 00000000`50c1fc58 00000000`00000002 : influxd+0x10e09
01 00000000`004343aa : 00000000`00101cf8 00000000`00000001 00000000`50c1fc58 00000000`00463333 : influxd+0x3446e
02 00000000`00433fff : 000000c0`01980248 00000000`50c1fcc8 00000000`00410f96 00000000`00000000 : influxd+0x343aa
03 00000000`00410f96 : 00000000`00000000 00000000`00000000 000000c0`00a861c0 00000000`00000000 : influxd+0x33fff
04 00000000`0044a94d : 000000c0`285c6000 00000001`00000002 00000000`00000000 00000000`00000000 : influxd+0x10f96
05 00000000`00459a71 : 000000c0`285c6000 00000000`50c1fd68 00000000`0042cd08 00000000`00000000 : influxd+0x4a94d
06 00000000`0047ea79 : 000000c0`01980060 00000000`00004000 000000c0`00a861c0 00000000`00000000 : influxd+0x59a71
07 000000c0`01980060 : 00000000`00004000 000000c0`00a861c0 00000000`00000000 00000000`0047e8a5 : influxd+0x7ea79
08 00000000`00004000 : 000000c0`00a861c0 00000000`00000000 00000000`0047e8a5 00000000`02907c11 : 0x000000c0`01980060
09 000000c0`00a861c0 : 00000000`00000000 00000000`0047e8a5 00000000`02907c11 00000000`00000000 : 0x4000
0a 00000000`00000000 : 00000000`0047e8a5 00000000`02907c11 00000000`00000000 00000000`00000000 : 0x000000c0`00a861c0

24 Id: 1998.2830 Suspend: 0 Teb: 00000000`0031f000 Unfrozen
RetAddr : Args to Child : Call Site
00 00007ffe`008bda3e : 000000c0`01a002e8 00000000`00000102 00000000`50e5f3a8 000000c0`01a00008 : ntdll!ZwWaitForSingleObject+0x14
01 00000000`004821c9 : 000000c0`01a002e8 00000000`0031f000 00000000`00000000 00000000`00000454 : KERNELBASE!WaitForSingleObjectEx+0x8e
02 00000000`0048054d : 00000000`50e5f800 00000000`00000b04 00000000`00000b04 00007ffe`00913a7b : influxd+0x821c9
03 00000000`00480601 : 00000000`50e5f5d0 00000000`00000000 00000000`50e5f608 00000000`0043c91b : influxd+0x8054d
......

25 Id: 1998.288c Suspend: 0 Teb: 00000000`00321000 Unfrozen
RetAddr : Args to Child : Call Site
00 00007ffe`008bda3e : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!ZwWaitForSingleObject+0x14
01 00000000`004821c9 : 000000c0`007792e8 00000000`00321000 00000000`00000000 00000000`000004d8 : KERNELBASE!WaitForSingleObjectEx+0x8e
02 00000000`0048054d : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : influxd+0x821c9
03 00000000`00480601 : 00000000`51b3fa30 00000000`00000000 00000000`51b3fa68 00000000`0043c91b : influxd+0x8054d
......

26 Id: 1998.2890 Suspend: 0 Teb: 00000000`00323000 Unfrozen
RetAddr : Args to Child : Call Site
00 00007ffe`008bda3e : 00000000`51d3fd40 00000000`0043d825 00000000`02907f90 00000000`0000099c : ntdll!ZwWaitForSingleObject+0x14
01 00000000`004821c9 : 000000c0`00701ae8 00000000`00323000 00000000`00000000 00000000`000004f0 : KERNELBASE!WaitForSingleObjectEx+0x8e
02 00000000`0048054d : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : influxd+0x821c9
03 00000000`00480601 : 00000000`51d3fa30 00000000`00000000 00000000`51d3fa68 00000000`0043c91b : influxd+0x8054d
......

27 Id: 1998.2894 Suspend: 0 Teb: 00000000`00325000 Unfrozen
RetAddr : Args to Child : Call Site
00 00007ffe`008bda3e : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!ZwWaitForSingleObject+0x14
01 00000000`004821c9 : 000000c0`01980ae8 00000000`00325000 00000000`00000000 00000000`000004d4 : KERNELBASE!WaitForSingleObjectEx+0x8e
02 00000000`0048054d : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : influxd+0x821c9
03 00000000`00480601 : 00000000`51f7fa30 00000000`00000000 00000000`51f7fa68 00000000`0043c91b : influxd+0x8054d
......

28 Id: 1998.2898 Suspend: 0 Teb: 00000000`00327000 Unfrozen
RetAddr : Args to Child : Call Site
00 00007ffe`008bda3e : 0000000b`0000000e 00000000`00000003 0002e617`d710d858 0002e628`86c775d0 : ntdll!ZwWaitForSingleObject+0x14
01 00000000`004821c9 : 000000c0`01a00ae8 00000000`00000000 00000000`00000000 00000000`00000514 : KERNELBASE!WaitForSingleObjectEx+0x8e
02 00000000`0048054d : 000000c0`01a00808 00000000`5217fe40 00000000`0043c979 00000000`5217fe50 : influxd+0x821c9
03 00000000`004805c3 : 00000000`000004e0 000000c0`02aa1a40 00000000`0047e9a9 000000c0`01a00860 : influxd+0x8054d
......

29 Id: 1998.28b4 Suspend: 0 Teb: 00000000`00329000 Unfrozen
RetAddr : Args to Child : Call Site
00 00007ffe`008bda3e : 00000000`062536c8 00000000`02c8e620 00000000`00325d7b 00000000`00bb126d : ntdll!ZwWaitForSingleObject+0x14
01 00000000`004821c9 : 000000c0`00acb2e8 00000000`00463333 00000000`00000000 00000000`00000650 : KERNELBASE!WaitForSingleObjectEx+0x8e
02 00000000`0048054d : 00000000`5404fcd8 00000000`004821c9 000000c0`00acb330 00000000`5404fd10 : influxd+0x821c9
03 00000000`00480601 : 00000000`5404fce8 00000000`00000000 00000000`5404fd10 00000000`5404fd20 : influxd+0x8054d
......

30 Id: 1998.28b8 Suspend: 0 Teb: 00000000`0032b000 Unfrozen
RetAddr : Args to Child : Call Site
00 00007ffe`008bda3e : 00000001`00000004 00000000`00000000 0002e617`d4b56c2c 00000000`00000000 : ntdll!ZwWaitForSingleObject+0x14
01 00000000`004821c9 : 000000c0`00779ae8 00000000`00000000 00000000`00000000 00000000`000005c8 : KERNELBASE!WaitForSingleObjectEx+0x8e
02 00000000`0048054d : 000000c0`00779808 00000000`5424fe40 00000000`0043c979 00000000`00000200 : influxd+0x821c9
03 00000000`004805c3 : 00000000`00000388 000000c0`23db4c40 00000000`0047e9a9 000000c0`00779860 : influxd+0x8054d
......

31 Id: 1998.28bc Suspend: 0 Teb: 00000000`0032d000 Unfrozen
RetAddr : Args to Child : Call Site
00 00007ffe`008bda3e : 00000000`00000000 00000000`00410f96 000000c0`01981008 00000000`5444f7e0 : ntdll!ZwWaitForSingleObject+0x14
01 00000000`004821c9 : 000000c0`019812e8 00000000`0032d000 00000000`00000000 00000000`000005cc : KERNELBASE!WaitForSingleObjectEx+0x8e
02 00000000`0048054d : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : influxd+0x821c9
03 00000000`00480601 : 00000000`5444fa00 00000000`00000000 00000000`5444fa38 00000000`0043c91b : influxd+0x8054d
......

32 Id: 1998.28c0 Suspend: 0 Teb: 00000000`0032f000 Unfrozen
RetAddr : Args to Child : Call Site
00 00007ffe`008bda3e : 00000000`5464fd40 00000000`0043d825 00000000`02907f90 00000000`00000d28 : ntdll!ZwWaitForSingleObject+0x14
01 00000000`004821c9 : 000000c0`01a012e8 00000000`0032f000 00000000`00000000 00000000`00000660 : KERNELBASE!WaitForSingleObjectEx+0x8e
02 00000000`0048054d : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : influxd+0x821c9
03 00000000`00480601 : 00000000`5464fa30 00000000`00000000 00000000`5464fa68 00000000`0043c91b : influxd+0x8054d
......

33 Id: 1998.28c4 Suspend: 0 Teb: 00000000`00331000 Unfrozen
RetAddr : Args to Child : Call Site
00 00007ffe`008bda3e : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!ZwWaitForSingleObject+0x14
01 00000000`004821c9 : 000000c0`006012e8 00000000`00000000 00000000`00000000 00000000`00000648 : KERNELBASE!WaitForSingleObjectEx+0x8e
02 00000000`0048054d : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : influxd+0x821c9
03 00000000`00480601 : 00000000`5484fb28 00000000`00000000 00000000`00000000 00000000`5484fb60 : influxd+0x8054d
......

34 Id: 1998.28c8 Suspend: 0 Teb: 00000000`00333000 Unfrozen
RetAddr : Args to Child : Call Site
00 00007ffe`008bda3e : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!ZwWaitForSingleObject+0x14
01 00000000`004821c9 : 000000c0`00580ae8 00000000`00000000 00000000`00000000 00000000`00000668 : KERNELBASE!WaitForSingleObjectEx+0x8e
02 00000000`0048054d : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : influxd+0x821c9
03 00000000`00480601 : 00000000`54a4fb28 00000000`00000000 00000000`00000000 00000000`54a4fb60 : influxd+0x8054d
......

35 Id: 1998.28d0 Suspend: 0 Teb: 00000000`00335000 Unfrozen
RetAddr : Args to Child : Call Site
00 00007ffe`008bda3e : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!ZwWaitForSingleObject+0x14
01 00000000`004821c9 : 000000c0`006812e8 00000000`00335000 00000000`00000000 00000000`000005f4 : KERNELBASE!WaitForSingleObjectEx+0x8e
02 00000000`0048054d : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : influxd+0x821c9
03 00000000`00480601 : 00000000`54e4fa30 00000000`00000000 00000000`54e4fa68 00000000`0043c91b : influxd+0x8054d
......

36 Id: 1998.28cc Suspend: 0 Teb: 00000000`00337000 Unfrozen
RetAddr : Args to Child : Call Site
00 00007ffe`008bda3e : 00000000`54c4fd40 00000000`0043d825 00000000`02907f90 00000000`00000cbc : ntdll!ZwWaitForSingleObject+0x14
01 00000000`004821c9 : 000000c0`01a01ae8 00000000`00337000 00000000`00000000 00000000`000005e8 : KERNELBASE!WaitForSingleObjectEx+0x8e
02 00000000`0048054d : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : influxd+0x821c9
03 00000000`00480601 : 00000000`54c4fa30 00000000`00000000 00000000`54c4fa68 00000000`0043c91b : influxd+0x8054d
......

37 Id: 1998.28d4 Suspend: 0 Teb: 00000000`00339000 Unfrozen
RetAddr : Args to Child : Call Site
00 00007ffe`008bda3e : 00000000`5504faf0 00007ffe`008ea66d 00000000`00463333 00000000`5504fb00 : ntdll!ZwWaitForSingleObject+0x14
01 00000000`004821c9 : 000000c0`031842e8 000000c0`257c9800 000000c0`00000000 00000000`000005c0 : KERNELBASE!WaitForSingleObjectEx+0x8e
02 00000000`0048054d : 000000c0`03184330 00000000`004821c9 00000000`004213a5 000000c0`03184008 : influxd+0x821c9
03 00000000`00480601 : 00000000`5504fce8 00000000`00000000 00000000`0043bdc5 00000000`5504fd20 : influxd+0x8054d
......

__Dump2:__
Catch dump file of one application which has been running only 1 days, analyze.

0:000> !runaway
User Mode Time
Thread Time
1:1e48 0 days 2:06:21.015
12:2208 0 days 0:04:31.703
26:274c 0 days 0:04:14.781
15:2214 0 days 0:04:13.187
27:be0 0 days 0:04:12.281
14:2210 0 days 0:04:11.390
28:27c4 0 days 0:04:09.000
16:2218 0 days 0:03:57.437
3:1e50 0 days 0:03:53.484
13:220c 0 days 0:03:51.781
2:1e4c 0 days 0:03:50.843
32:2948 0 days 0:03:47.296
4:1e54 0 days 0:03:46.593
18:2220 0 days 0:03:44.281
17:221c 0 days 0:03:44.203
19:2224 0 days 0:03:40.078
22:2230 0 days 0:03:38.859
8:21f8 0 days 0:03:32.875
6:21f0 0 days 0:03:29.000
7:21f4 0 days 0:03:21.656
20:2228 0 days 0:03:21.515
21:222c 0 days 0:03:18.093
33:294c 0 days 0:03:16.828
9:21fc 0 days 0:03:09.281
31:2940 0 days 0:03:04.218
34:2950 0 days 0:03:00.078
23:2234 0 days 0:02:55.375
30:293c 0 days 0:02:53.468
25:2248 0 days 0:02:35.921
24:2238 0 days 0:02:28.640
11:2204 0 days 0:02:24.031
10:2200 0 days 0:02:23.906
29:27e4 0 days 0:02:03.359
0:1340 0 days 0:00:00.109
35:e2c 0 days 0:00:00.000
5:1e5c 0 days 0:00:00.000

Still can find one thread is not wait for single object as below:
26 Id: 133c.274c Suspend: 0 Teb: 00000000`00322000 Unfrozen
RetAddr : Args to Child : Call Site
00 00000000`0043436b : 00000000`000001f0 00000000`00000001 000000c0`00504d80 00000000`00463333 : influxd+0x344af
01 00000000`00433fff : 000000c0`01900248 00000000`5123fcc8 00000000`00410f96 00000000`00000000 : influxd+0x3436b
02 00000000`00410f96 : 00000000`00000000 00000000`00000000 000000c0`00a861c0 00000000`00000000 : influxd+0x33fff
03 00000000`0044a94d : 000000c0`0f480700 00000001`00000002 00000000`00000000 00000000`00000000 : influxd+0x10f96
04 00000000`00459a71 : 000000c0`0f480700 00000000`00000000 00000000`00000000 00000000`00000000 : influxd+0x4a94d
05 00000000`0047ea79 : 000000c0`01900060 00000000`00004000 000000c0`00a861c0 00000000`00000000 : influxd+0x59a71
06 000000c0`01900060 : 00000000`00004000 000000c0`00a861c0 00000000`00000000 00000000`0047e8a5 : influxd+0x7ea79
07 00000000`00004000 : 000000c0`00a861c0 00000000`00000000 00000000`0047e8a5 00000000`02907c11 : 0x000000c0`01900060
08 000000c0`00a861c0 : 00000000`00000000 00000000`0047e8a5 00000000`02907c11 00000000`00000000 : 0x4000
09 00000000`00000000 : 00000000`0047e8a5 00000000`02907c11 00000000`00000000 00000000`00000000 : 0x000000c0`00a861c0

I will paste all call stack at below.

Contributor guide

Open the contributing guide

Research direction

Start with the supplied Windows dump, the influxd.exe process, and the !runaway/~*kb output; the report notes that symbols are unavailable. Check the /api/v2/orgs and ping endpoints while the process is hung, then establish a reproducible cause and a way to verify that those endpoints remain responsive.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.