node-exporter retrying every second

Today, we had a Jenkins outage. During the outage, I checked /var/log/messages, we have tons of these messages and updating continiously

Nov 18 14:55:36 jenkinscid-127_0_0_1 node-exporter[998]: 2018/11/18 14:55:36 http: Accept error: accept tcp [::]:9100: accept4: too many open files; retrying in 1s
Nov 18 14:55:37 jenkinscid-127_0_0_1 node-exporter[998]: 2018/11/18 14:55:37 http: Accept error: accept tcp [::]:9100: accept4: too many open files; retrying in 1s
Nov 18 14:55:38 jenkinscid-127_0_0_1 node-exporter[998]: 2018/11/18 14:55:38 http: Accept error: accept tcp [::]:9100: accept4: too many open files; retrying in 1s
Nov 18 14:55:39 jenkinscid-127_0_0_1 node-exporter[998]: 2018/11/18 14:55:39 http: Accept error: accept tcp [::]:9100: accept4: too many open files; retrying in 1s
Nov 18 14:55:40 jenkinscid-127_0_0_1 node-exporter[998]: 2018/11/18 14:55:40 http: Accept error: accept tcp [::]:9100: accept4: too many open files; retrying in 1s
Nov 18 14:55:41 jenkinscid-127_0_0_1 node-exporter[998]: 2018/11/18 14:55:41 http: Accept error: accept tcp [::]:9100: accept4: too many open files; retrying in 1s
Nov 18 14:55:42 jenkinscid-127_0_0_1 node-exporter[998]: 2018/11/18 14:55:42 http: Accept error: accept tcp [::]:9100: accept4: too many open files; retrying in 1s

Then I ran lsof and sorted it to find out who the topmost processes that have open files

COUNT PID PROCESS
14247 998 node_expo
 544 12005 nginx
 496 10381 python
 304 898 amazon-ss
 240 633 gssproxy
 209 1014 collectd
 144 23672 sshd
 144 21935 sshd
 140 23666 sshd
 140 21930 sshd
 102 511 sssd_be
  99 10417 python
  99 10410 python
  99 10408 python
  99 10402 python
  99 10395 python
  99 10393 python
  97 10390 python
  96 470 sssd
  95 603 sssd_nss

We're using AWS EFS. I was thinking that our NFS mount went away. However, running cat /proc/mounts still showed it was mounted. I can still netcat to port 2049. However, I was unable to run ls /var/lib/jenkins. When I stopped the jenkins service, that was the only time I was able to access /var/lib/jenkins.

Kernel is 4.16.11-100.fc26.x86_64 #1 SMP Tue May 22 20:02:12 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Output of cat /proc/version is

Linux version 4.16.11-100.fc26.x86_64 (mockbuild@bkernel02.phx2.fedoraproject.org) (gcc version 7.3.1 20180130 (Red Hat 7.3.1-2) (GCC)) #1 SMP Tue May 22 20:02:12 UTC 2018

node_exporter is

node_exporter, version 0.16.0 (branch: HEAD, revision: d42bd70f4363dced6b77d8fc311ea57b63387e4f)
  build user:       root@a67a9bc13a69
  build date:       20180515-15:52:42
  go version:       go1.9.6

ulimit -a

core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 62211
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 62211
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Start by reviewing the node_exporter 0.16.0 behavior described in the report, especially the repeated accept errors under the open-files limit of 1024. Reproduce or trace the failure with the AWS EFS and Jenkins conditions described, then determine what behavior should change and verify that retries no longer create the reported log flood.