saltstack / saltstack/salt

[BUG] connection between consul and salt-master not terminating

Open
#59,283 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug severity-medium
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description
We use consul for storing pillar configuration... We're using consul 1.8.5, python-consul 1.1.0 and salt-master 3002.2-1
We are noticing stale connections between salt-master worker threads and consul and this keeps growing.

ps aux | grep -i [M]Worker- 
root       538  0.0  0.0 569828 83864 ?        Sl   Jan07   1:19 /usr/bin/python3.6 /usr/bin/salt-master MWorker-0
root       539  0.0  0.0 579128 89756 ?        Sl   Jan07   1:21 /usr/bin/python3.6 /usr/bin/salt-master MWorker-1
root       540  0.0  0.0 578144 89500 ?        Sl   Jan07   1:18 /usr/bin/python3.6 /usr/bin/salt-master MWorker-2
root       541  0.0  0.0 570304 83704 ?        Sl   Jan07   1:19 /usr/bin/python3.6 /usr/bin/salt-master MWorker-3
root       542  0.0  0.0 573464 86892 ?        Sl   Jan07   1:18 /usr/bin/python3.6 /usr/bin/salt-master MWorker-4
root       545  0.0  0.0 579744 89880 ?        Sl   Jan07   1:19 /usr/bin/python3.6 /usr/bin/salt-master MWorker-5
root       546  0.0  0.0 573144 87516 ?        Sl   Jan07   1:19 /usr/bin/python3.6 /usr/bin/salt-master MWorker-6
root       547  0.0  0.0 578532 89108 ?        Sl   Jan07   1:20 /usr/bin/python3.6 /usr/bin/salt-master MWorker-7
netstat -ntp | grep 8500 | grep -v consul | grep -E '53[89]|54[012567]/salt-master' 
tcp        0      0 127.0.0.1:58078         127.0.0.1:8500          ESTABLISHED 540/salt-master MWo
tcp        0      0 127.0.0.1:44788         127.0.0.1:8500          ESTABLISHED 541/salt-master MWo
tcp        0      0 127.0.0.1:53802         127.0.0.1:8500          ESTABLISHED 539/salt-master MWo
tcp        0      0 127.0.0.1:42438         127.0.0.1:8500          ESTABLISHED 546/salt-master MWo
tcp        0      0 127.0.0.1:37010         127.0.0.1:8500          ESTABLISHED 545/salt-master MWo
tcp        0      0 127.0.0.1:46810         127.0.0.1:8500          ESTABLISHED 545/salt-master MWo
tcp        0      0 127.0.0.1:33702         127.0.0.1:8500          ESTABLISHED 538/salt-master MWo
tcp        0      0 127.0.0.1:56044         127.0.0.1:8500          ESTABLISHED 539/salt-master MWo
tcp        0      0 127.0.0.1:60390         127.0.0.1:8500          ESTABLISHED 545/salt-master MWo
tcp        0      0 127.0.0.1:39326         127.0.0.1:8500          ESTABLISHED 545/salt-master MWo
tcp        0      0 127.0.0.1:51624         127.0.0.1:8500          ESTABLISHED 538/salt-master MWo
tcp        0      0 127.0.0.1:50346         127.0.0.1:8500          ESTABLISHED 542/salt-master MWo
tcp        0      0 127.0.0.1:44160         127.0.0.1:8500          ESTABLISHED 542/salt-master MWo
tcp        0      0 127.0.0.1:40896         127.0.0.1:8500          ESTABLISHED 545/salt-master MWo
tcp        0      0 127.0.0.1:40190         127.0.0.1:8500          ESTABLISHED 547/salt-master MWo
tcp        0      0 127.0.0.1:41252         127.0.0.1:8500          ESTABLISHED 540/salt-master MWo
tcp        0      0 127.0.0.1:45426         127.0.0.1:8500          ESTABLISHED 547/salt-master MWo
tcp        0      0 127.0.0.1:36992         127.0.0.1:8500          ESTABLISHED 546/salt-master MWo
tcp        0      0 127.0.0.1:33540         127.0.0.1:8500          ESTABLISHED 547/salt-master MWo
tcp        0      0 127.0.0.1:38918         127.0.0.1:8500          ESTABLISHED 538/salt-master MWo
tcp        0      0 127.0.0.1:59800         127.0.0.1:8500          ESTABLISHED 540/salt-master MWo
tcp        0      0 127.0.0.1:36834         127.0.0.1:8500          ESTABLISHED 541/salt-master MWo
tcp        0      0 127.0.0.1:57176         127.0.0.1:8500          ESTABLISHED 539/salt-master MWo
tcp        0      0 127.0.0.1:54674         127.0.0.1:8500          ESTABLISHED 545/salt-master MWo
tcp        0      0 127.0.0.1:56686         127.0.0.1:8500          ESTABLISHED 540/salt-master MWo
tcp        0      0 127.0.0.1:43250         127.0.0.1:8500          ESTABLISHED 539/salt-master MWo
tcp        0      0 127.0.0.1:51318         127.0.0.1:8500          ESTABLISHED 540/salt-master MWo
tcp        0      0 127.0.0.1:59830         127.0.0.1:8500          ESTABLISHED 541/salt-master MWo
tcp        0      0 127.0.0.1:43212         127.0.0.1:8500          ESTABLISHED 542/salt-master MWo
tcp        0      0 127.0.0.1:46882         127.0.0.1:8500          ESTABLISHED 547/salt-master MWo
tcp        0      0 127.0.0.1:37660         127.0.0.1:8500          ESTABLISHED 545/salt-master MWo
tcp        0      0 127.0.0.1:58332         127.0.0.1:8500          ESTABLISHED 546/salt-master MWo
tcp        0      0 127.0.0.1:50166         127.0.0.1:8500          ESTABLISHED 538/salt-master MWo
tcp        0      0 127.0.0.1:58080         127.0.0.1:8500          ESTABLISHED 540/salt-master MWo
tcp        0      0 127.0.0.1:46860         127.0.0.1:8500          ESTABLISHED 541/salt-master MWo
tcp        0      0 127.0.0.1:47548         127.0.0.1:8500          ESTABLISHED 547/salt-master MWo
tcp        0      0 127.0.0.1:50026         127.0.0.1:8500          ESTABLISHED 541/salt-master MWo
tcp        0      0 127.0.0.1:42182         127.0.0.1:8500          ESTABLISHED 542/salt-master MWo
tcp        0      0 127.0.0.1:46020         127.0.0.1:8500          ESTABLISHED 538/salt-master MWo
tcp        0      0 127.0.0.1:50164         127.0.0.1:8500          ESTABLISHED 538/salt-master MWo
tcp        0      0 127.0.0.1:41520         127.0.0.1:8500          ESTABLISHED 538/salt-master MWo
tcp        0      0 127.0.0.1:58544         127.0.0.1:8500          ESTABLISHED 546/salt-master MWo
tcp        0      0 127.0.0.1:49964         127.0.0.1:8500          ESTABLISHED 540/salt-master MWo
tcp        0      0 127.0.0.1:53248         127.0.0.1:8500          ESTABLISHED 539/salt-master MWo
tcp        0      0 127.0.0.1:50154         127.0.0.1:8500          ESTABLISHED 541/salt-master MWo
tcp        0      0 127.0.0.1:36466         127.0.0.1:8500          ESTABLISHED 539/salt-master MWo
tcp        0      0 127.0.0.1:60324         127.0.0.1:8500          ESTABLISHED 547/salt-master MWo
tcp        0      0 127.0.0.1:49266         127.0.0.1:8500          ESTABLISHED 547/salt-master MWo
tcp        0      0 127.0.0.1:49150         127.0.0.1:8500          ESTABLISHED 546/salt-master MWo
tcp        0      0 127.0.0.1:59784         127.0.0.1:8500          ESTABLISHED 539/salt-master MWo
tcp        0      0 127.0.0.1:58444         127.0.0.1:8500          ESTABLISHED 541/salt-master MWo
tcp        0      0 127.0.0.1:54006         127.0.0.1:8500          ESTABLISHED 541/salt-master MWo
tcp        0      0 127.0.0.1:53534         127.0.0.1:8500          ESTABLISHED 545/salt-master MWo
tcp        0      0 127.0.0.1:39102         127.0.0.1:8500          ESTABLISHED 539/salt-master MWo
tcp        0      0 127.0.0.1:39328         127.0.0.1:8500          ESTABLISHED 545/salt-master MWo
tcp        0      0 127.0.0.1:45276         127.0.0.1:8500          ESTABLISHED 541/salt-master MWo
tcp        0      0 127.0.0.1:50620         127.0.0.1:8500          ESTABLISHED 540/salt-master MWo
tcp        0      0 127.0.0.1:49734         127.0.0.1:8500          ESTABLISHED 545/salt-master MWo
tcp        0      0 127.0.0.1:45424         127.0.0.1:8500          ESTABLISHED 547/salt-master MWo
tcp        0      0 127.0.0.1:47206         127.0.0.1:8500          ESTABLISHED 541/salt-master MWo
tcp        0      0 127.0.0.1:50156         127.0.0.1:8500          ESTABLISHED 541/salt-master MWo
tcp        0      0 127.0.0.1:43432         127.0.0.1:8500          ESTABLISHED 542/salt-master MWo
tcp        0      0 127.0.0.1:47378         127.0.0.1:8500          ESTABLISHED 546/salt-master MWo
tcp        0      0 127.0.0.1:43792         127.0.0.1:8500          ESTABLISHED 538/salt-master MWo
tcp        0      0 127.0.0.1:57704         127.0.0.1:8500          ESTABLISHED 546/salt-master MWo
tcp        0      0 127.0.0.1:49832         127.0.0.1:8500          ESTABLISHED 541/salt-master MWo
tcp        0      0 127.0.0.1:37486         127.0.0.1:8500          ESTABLISHED 541/salt-master MWo
tcp        0      0 127.0.0.1:45362         127.0.0.1:8500          ESTABLISHED 542/salt-master MWo
ll -lrt /proc/$(pgrep consul)/fd/ | grep socket 
lrwx------. 1 consul consul 64 Jan  7 14:13 94 -> socket:[227596252]
lrwx------. 1 consul consul 64 Jan  7 14:13 92 -> socket:[227348019]
lrwx------. 1 consul consul 64 Jan  7 14:13 9 -> socket:[203005177]
lrwx------. 1 consul consul 64 Jan  7 14:13 85 -> socket:[227597835]
lrwx------. 1 consul consul 64 Jan  7 14:13 84 -> socket:[227203422]
lrwx------. 1 consul consul 64 Jan  7 14:13 83 -> socket:[226812563]
lrwx------. 1 consul consul 64 Jan  7 14:13 82 -> socket:[226054346]
lrwx------. 1 consul consul 64 Jan  7 14:13 80 -> socket:[226220292]
lrwx------. 1 consul consul 64 Jan  7 14:13 8 -> socket:[203005176]
lrwx------. 1 consul consul 64 Jan  7 14:13 79 -> socket:[227734027]
lrwx------. 1 consul consul 64 Jan  7 14:13 78 -> socket:[224428638]
lrwx------. 1 consul consul 64 Jan  7 14:13 77 -> socket:[227806947]
lrwx------. 1 consul consul 64 Jan  7 14:13 76 -> socket:[227743543]
lrwx------. 1 consul consul 64 Jan  7 14:13 75 -> socket:[227734058]
lrwx------. 1 consul consul 64 Jan  7 14:13 74 -> socket:[226943419]
lrwx------. 1 consul consul 64 Jan  7 14:13 73 -> socket:[227707066]
lrwx------. 1 consul consul 64 Jan  7 14:13 72 -> socket:[226217161]
lrwx------. 1 consul consul 64 Jan  7 14:13 71 -> socket:[227341645]
lrwx------. 1 consul consul 64 Jan  7 14:13 70 -> socket:[225843276]
lrwx------. 1 consul consul 64 Jan  7 14:13 67 -> socket:[217933389]
lrwx------. 1 consul consul 64 Jan  7 14:13 62 -> socket:[223650690]
lrwx------. 1 consul consul 64 Jan  7 14:13 6 -> socket:[202982812]
lrwx------. 1 consul consul 64 Jan  7 14:13 49 -> socket:[218039800]
lrwx------. 1 consul consul 64 Jan  7 14:13 45 -> socket:[216489253]
lrwx------. 1 consul consul 64 Jan  7 14:13 43 -> socket:[227034376]
lrwx------. 1 consul consul 64 Jan  7 14:13 40 -> socket:[221195726]
lrwx------. 1 consul consul 64 Jan  7 14:13 38 -> socket:[212850114]
lrwx------. 1 consul consul 64 Jan  7 14:13 35 -> socket:[216292684]
lrwx------. 1 consul consul 64 Jan  7 14:13 34 -> socket:[216635853]
lrwx------. 1 consul consul 64 Jan  7 14:13 25 -> socket:[215403348]
lrwx------. 1 consul consul 64 Jan  7 14:13 24 -> socket:[203049949]
lrwx------. 1 consul consul 64 Jan  7 14:13 23 -> socket:[210197484]
lrwx------. 1 consul consul 64 Jan  7 14:13 22 -> socket:[203048656]
lrwx------. 1 consul consul 64 Jan  7 14:13 21 -> socket:[205917858]
lrwx------. 1 consul consul 64 Jan  7 14:13 20 -> socket:[204523052]
lrwx------. 1 consul consul 64 Jan  7 14:13 2 -> socket:[202975993]
lrwx------. 1 consul consul 64 Jan  7 14:13 19 -> socket:[203043270]
lrwx------. 1 consul consul 64 Jan  7 14:13 18 -> socket:[202993046]
lrwx------. 1 consul consul 64 Jan  7 14:13 17 -> socket:[202981708]
lrwx------. 1 consul consul 64 Jan  7 14:13 16 -> socket:[203001928]
lrwx------. 1 consul consul 64 Jan  7 14:13 15 -> socket:[203001286]
lrwx------. 1 consul consul 64 Jan  7 14:13 14 -> socket:[202946850]
lrwx------. 1 consul consul 64 Jan  7 14:13 13 -> socket:[203005179]
lrwx------. 1 consul consul 64 Jan  7 14:13 12 -> socket:[203005178]
lrwx------. 1 consul consul 64 Jan  7 14:13 1 -> socket:[202975993]
lrwx------. 1 consul consul 64 Jan  7 14:41 51 -> socket:[225965122]
lrwx------. 1 consul consul 64 Jan  7 14:41 50 -> socket:[223465929]
lrwx------. 1 consul consul 64 Jan  7 14:41 48 -> socket:[209724893]
lrwx------. 1 consul consul 64 Jan  7 14:41 47 -> socket:[203336994]
lrwx------. 1 consul consul 64 Jan  7 14:41 46 -> socket:[225479342]
lrwx------. 1 consul consul 64 Jan  7 14:41 44 -> socket:[212855474]
lrwx------. 1 consul consul 64 Jan  7 14:41 42 -> socket:[213899469]
lrwx------. 1 consul consul 64 Jan  7 14:41 41 -> socket:[212612339]
lrwx------. 1 consul consul 64 Jan  7 14:41 39 -> socket:[208184151]
lrwx------. 1 consul consul 64 Jan  7 14:41 37 -> socket:[208533737]
lrwx------. 1 consul consul 64 Jan  7 14:41 36 -> socket:[211600685]
lrwx------. 1 consul consul 64 Jan  7 14:41 33 -> socket:[214080943]
lrwx------. 1 consul consul 64 Jan  7 14:41 32 -> socket:[203049442]
lrwx------. 1 consul consul 64 Jan  7 14:41 31 -> socket:[206565673]
lrwx------. 1 consul consul 64 Jan  7 14:41 30 -> socket:[203414012]
lrwx------. 1 consul consul 64 Jan  7 14:41 29 -> socket:[219668079]
lrwx------. 1 consul consul 64 Jan  7 14:41 28 -> socket:[208159361]
lrwx------. 1 consul consul 64 Jan  7 14:41 27 -> socket:[226435590]
lrwx------. 1 consul consul 64 Jan  7 14:41 26 -> socket:[207351858]
lrwx------. 1 consul consul 64 Jan  7 14:45 52 -> socket:[223088204]
lrwx------. 1 consul consul 64 Jan  9 12:14 63 -> socket:[214494159]
lrwx------. 1 consul consul 64 Jan  9 12:14 61 -> socket:[220073131]
lrwx------. 1 consul consul 64 Jan  9 12:14 60 -> socket:[214933464]
lrwx------. 1 consul consul 64 Jan  9 12:14 59 -> socket:[222639128]
lrwx------. 1 consul consul 64 Jan  9 12:14 58 -> socket:[227686391]
lrwx------. 1 consul consul 64 Jan  9 12:14 57 -> socket:[209913739]
lrwx------. 1 consul consul 64 Jan  9 12:14 56 -> socket:[214871728]
lrwx------. 1 consul consul 64 Jan  9 12:14 55 -> socket:[211058071]
lrwx------. 1 consul consul 64 Jan  9 12:14 54 -> socket:[225817765]
lrwx------. 1 consul consul 64 Jan  9 12:14 53 -> socket:[218160706]
lrwx------. 1 consul consul 64 Jan  9 12:15 64 -> socket:[227807263]
lrwx------. 1 consul consul 64 Jan 11 23:14 69 -> socket:[226631058]
lrwx------. 1 consul consul 64 Jan 11 23:14 68 -> socket:[227327164]
lrwx------. 1 consul consul 64 Jan 11 23:14 66 -> socket:[223895801]
lrwx------. 1 consul consul 64 Jan 11 23:14 65 -> socket:[227179403]
lsof -p $(pgrep consul) | grep IPv4 | grep ESTAB 
consul  1658 consul   17u     IPv4          202981708      0t0       TCP localhost:41287->localhost:tmi (ESTABLISHED)
consul  1658 consul   18u     IPv4          202993046      0t0       TCP localhost:tmi->localhost:41287 (ESTABLISHED)
consul  1658 consul   19u     IPv4          203043270      0t0       TCP localhost:fmtp->localhost:47378 (ESTABLISHED)
consul  1658 consul   20u     IPv4          204523052      0t0       TCP localhost:fmtp->localhost:51624 (ESTABLISHED)
consul  1658 consul   21u     IPv4          205917858      0t0       TCP localhost:fmtp->localhost:37660 (ESTABLISHED)
consul  1658 consul   22u     IPv4          203048656      0t0       TCP localhost:fmtp->localhost:46882 (ESTABLISHED)
consul  1658 consul   23u     IPv4          210197484      0t0       TCP localhost:fmtp->localhost:53534 (ESTABLISHED)
consul  1658 consul   24u     IPv4          203049949      0t0       TCP localhost:fmtp->localhost:47548 (ESTABLISHED)
consul  1658 consul   25u     IPv4          215403348      0t0       TCP localhost:fmtp->localhost:49266 (ESTABLISHED)
consul  1658 consul   26u     IPv4          207351858      0t0       TCP localhost:fmtp->localhost:58544 (ESTABLISHED)
consul  1658 consul   27u     IPv4          226435590      0t0       TCP localhost:fmtp->localhost:33540 (ESTABLISHED)
consul  1658 consul   28u     IPv4          208159361      0t0       TCP localhost:fmtp->localhost:49964 (ESTABLISHED)
consul  1658 consul   29u     IPv4          219668079      0t0       TCP localhost:fmtp->localhost:45276 (ESTABLISHED)
consul  1658 consul   30u     IPv4          203414012      0t0       TCP localhost:fmtp->localhost:40190 (ESTABLISHED)
consul  1658 consul   31u     IPv4          206565673      0t0       TCP localhost:fmtp->localhost:43212 (ESTABLISHED)
consul  1658 consul   32u     IPv4          203049442      0t0       TCP localhost:fmtp->localhost:46860 (ESTABLISHED)
consul  1658 consul   33u     IPv4          214080943      0t0       TCP localhost:fmtp->localhost:36834 (ESTABLISHED)
consul  1658 consul   34u     IPv4          216635853      0t0       TCP localhost:fmtp->localhost:37486 (ESTABLISHED)
consul  1658 consul   35u     IPv4          216292684      0t0       TCP localhost:fmtp->localhost:39102 (ESTABLISHED)
consul  1658 consul   36u     IPv4          211600685      0t0       TCP localhost:fmtp->localhost:49150 (ESTABLISHED)
consul  1658 consul   37u     IPv4          208533737      0t0       TCP localhost:fmtp->localhost:43792 (ESTABLISHED)
consul  1658 consul   38u     IPv4          212850114      0t0       TCP localhost:fmtp->localhost:60324 (ESTABLISHED)
consul  1658 consul   39u     IPv4          208184151      0t0       TCP localhost:fmtp->localhost:50026 (ESTABLISHED)
consul  1658 consul   40u     IPv4          221195726      0t0       TCP localhost:fmtp->localhost:41520 (ESTABLISHED)
consul  1658 consul   41u     IPv4          212612339      0t0       TCP localhost:fmtp->localhost:43250 (ESTABLISHED)
consul  1658 consul   42u     IPv4          213899469      0t0       TCP localhost:fmtp->localhost:57176 (ESTABLISHED)
consul  1658 consul   43u     IPv4          227034376      0t0       TCP localhost:fmtp->localhost:36466 (ESTABLISHED)
consul  1658 consul   44u     IPv4          212855474      0t0       TCP localhost:fmtp->localhost:60390 (ESTABLISHED)
consul  1658 consul   45u     IPv4          216489253      0t0       TCP localhost:fmtp->localhost:50346 (ESTABLISHED)
consul  1658 consul   46u     IPv4          225479342      0t0       TCP localhost:fmtp->localhost:44160 (ESTABLISHED)
consul  1658 consul   47u     IPv4          203336994      0t0       TCP localhost:fmtp->localhost:33702 (ESTABLISHED)
consul  1658 consul   48u     IPv4          209724893      0t0       TCP localhost:fmtp->localhost:54006 (ESTABLISHED)
consul  1658 consul   49u     IPv4          218039800      0t0       TCP localhost:fmtp->localhost:44788 (ESTABLISHED)
consul  1658 consul   50u     IPv4          223465929      0t0       TCP localhost:fmtp->localhost:49734 (ESTABLISHED)
consul  1658 consul   51u     IPv4          225965122      0t0       TCP localhost:fmtp->localhost:37010 (ESTABLISHED)
consul  1658 consul   52u     IPv4          223088204      0t0       TCP localhost:fmtp->localhost:56044 (ESTABLISHED)
consul  1658 consul   53u     IPv4          218160706      0t0       TCP localhost:fmtp->localhost:53248 (ESTABLISHED)
consul  1658 consul   54u     IPv4          225817765      0t0       TCP localhost:fmtp->localhost:58332 (ESTABLISHED)
consul  1658 consul   55u     IPv4          211058071      0t0       TCP localhost:fmtp->localhost:46020 (ESTABLISHED)
consul  1658 consul   56u     IPv4          214871728      0t0       TCP localhost:fmtp->localhost:49832 (ESTABLISHED)
consul  1658 consul   57u     IPv4          209913739      0t0       TCP localhost:fmtp->localhost:36992 (ESTABLISHED)
consul  1658 consul   58u     IPv4          227686391      0t0       TCP localhost:fmtp->localhost:45424 (ESTABLISHED)
consul  1658 consul   59u     IPv4          222639128      0t0       TCP localhost:fmtp->localhost:59800 (ESTABLISHED)
consul  1658 consul   60u     IPv4          214933464      0t0       TCP localhost:fmtp->localhost:53802 (ESTABLISHED)
consul  1658 consul   61u     IPv4          220073131      0t0       TCP localhost:fmtp->localhost:38918 (ESTABLISHED)
consul  1658 consul   62u     IPv4          223650690      0t0       TCP localhost:fmtp->localhost:59830 (ESTABLISHED)
consul  1658 consul   63u     IPv4          214494159      0t0       TCP localhost:fmtp->localhost:57704 (ESTABLISHED)
consul  1658 consul   64u     IPv4          227807263      0t0       TCP localhost:fmtp->localhost:50164 (ESTABLISHED)
consul  1658 consul   65u     IPv4          227179403      0t0       TCP localhost:fmtp->localhost:47206 (ESTABLISHED)
consul  1658 consul   66u     IPv4          223895801      0t0       TCP localhost:fmtp->localhost:41252 (ESTABLISHED)
consul  1658 consul   67u     IPv4          217933389      0t0       TCP localhost:fmtp->localhost:42182 (ESTABLISHED)
consul  1658 consul   68u     IPv4          227327164      0t0       TCP localhost:fmtp->localhost:56686 (ESTABLISHED)
consul  1658 consul   69u     IPv4          226631058      0t0       TCP localhost:fmtp->localhost:43432 (ESTABLISHED)
consul  1658 consul   70u     IPv4          225843276      0t0       TCP localhost:fmtp->localhost:58444 (ESTABLISHED)
consul  1658 consul   71u     IPv4          227341645      0t0       TCP localhost:fmtp->localhost:58078 (ESTABLISHED)
consul  1658 consul   72u     IPv4          226217161      0t0       TCP localhost:fmtp->localhost:50620 (ESTABLISHED)
consul  1658 consul   73u     IPv4          227707066      0t0       TCP localhost:fmtp->localhost:45362 (ESTABLISHED)
consul  1658 consul   74u     IPv4          226943419      0t0       TCP localhost:fmtp->localhost:59784 (ESTABLISHED)
consul  1658 consul   75u     IPv4          227734058      0t0       TCP localhost:fmtp->localhost:50166 (ESTABLISHED)
consul  1658 consul   76u     IPv4          227743543      0t0       TCP localhost:fmtp->localhost:45426 (ESTABLISHED)
consul  1658 consul   77u     IPv4          227806947      0t0       TCP localhost:fmtp->localhost:50154 (ESTABLISHED)
consul  1658 consul   78u     IPv4          224428638      0t0       TCP localhost:fmtp->localhost:42438 (ESTABLISHED)
consul  1658 consul   79u     IPv4          227734027      0t0       TCP localhost:fmtp->localhost:50156 (ESTABLISHED)
consul  1658 consul   80u     IPv4          226220292      0t0       TCP localhost:fmtp->localhost:51318 (ESTABLISHED)
consul  1658 consul   82u     IPv4          226054346      0t0       TCP localhost:fmtp->localhost:40896 (ESTABLISHED)
consul  1658 consul   83u     IPv4          226812563      0t0       TCP localhost:fmtp->localhost:54674 (ESTABLISHED)
consul  1658 consul   84u     IPv4          227203422      0t0       TCP localhost:fmtp->localhost:46810 (ESTABLISHED)
consul  1658 consul   85u     IPv4          227597835      0t0       TCP localhost:fmtp->localhost:39326 (ESTABLISHED)
consul  1658 consul   92u     IPv4          227348019      0t0       TCP localhost:fmtp->localhost:58080 (ESTABLISHED)
consul  1658 consul   94u     IPv4          227596252      0t0       TCP localhost:fmtp->localhost:39328 (ESTABLISHED)

Setup
cat /etc/salt/master.d/consul.conf

consul_config:
  consul.host: 127.0.0.1
  consul.port: 8500

cat /etc/salt/master.d/pillar.conf

ext_pillar:
  - consul: consul_config root=salt-shared
  - consul: consul_config root=salt-private/%(minion_id)s

Steps to Reproduce the behavior
Overtime connections between consul and salt-master MWorker threads remain open ... eventually consul blocks new connection request. The only way to recover from this situation is to restart salt-master or consul.

Expected behavior
After use of connection eventually the connection will be closed.

Versions Report

salt --versions-report (Provided by running salt --versions-report. Please also mention any differences in master/minion versions.) 
salt --versions-report
Salt Version:
          Salt: 3002.2

Dependency Versions:
          cffi: 1.14.4
      cherrypy: unknown
      dateutil: 2.6.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 2.10.1
       libgit2: Not Installed
      M2Crypto: 0.33.0
          Mako: 1.0.6
       msgpack: 1.0.0
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.14
      pycrypto: Not Installed
  pycryptodome: 3.9.7
        pygit2: Not Installed
        Python: 3.6.8 (default, Nov 21 2019, 19:31:34)
  python-gnupg: Not Installed
        PyYAML: 3.12
         PyZMQ: 17.0.0
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.2

System Versions:
          dist: centos 8 Core
        locale: UTF-8
       machine: x86_64
       release: 4.18.0-147.el8.x86_64
        system: Linux
       version: CentOS Linux 8 Core

Additional context
Add any other context about the problem here.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Begin with the salt-master worker processes and the Consul connection behavior described for consul 1.8.5, python-consul 1.1.0, and salt-master 3002.2-1; reproduce while monitoring the ESTABLISHED connections on port 8500. Done means connections no longer accumulate as worker activity continues.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.