concurrent map read and map write in influx 1.7.3 version

__Actual behavior:__
standalone Influx is running for 30 days in a container without any issues, but suddenly crashed with "concurrent map read and map write" error

__Environment info:__

* System info:
Linux 4.19.81-OpenNetworkLinux x86_64
* bash-4.4# influxd version
InfluxDB v1.7.3 (git: 1.7 698dbc789aff13c2678357a6b93ff73dd7136571)

* Docker version 19.03.8 and k8 version is v.1.16.3

__Logs:__
ts=2020-12-02T02:08:34.532541Z lvl=info msg="Retention policy deletion check (start)" log_id=0Q6_M9EW000 service=retention trace_id=0QpJB2u0000 op_name=retention_delete_check op_event=start
ts=2020-12-02T02:08:34.550387Z lvl=info msg="Deleted shard group" log_id=0Q6_M9EW000 service=retention trace_id=0QpJB2u0000 op_name=retention_delete_check db_instance=auditlogs db_shard_group=4138 db_rp=auditLogs
ts=2020-12-02T02:08:34.566848Z lvl=info msg="Deleted shard group" log_id=0Q6_M9EW000 service=retention trace_id=0QpJB2u0000 op_name=retention_delete_check db_instance=Alarms db_shard_group=4139 db_rp=alarm
ts=2020-12-02T02:08:34.584970Z lvl=info msg="Deleted shard group" log_id=0Q6_M9EW000 service=retention trace_id=0QpJB2u0000 op_name=retention_delete_check db_instance=ActiveAlarms db_shard_group=4140 db_rp=alarm
ts=2020-12-02T02:08:34.603956Z lvl=info msg="Deleted shard group" log_id=0Q6_M9EW000 service=retention trace_id=0QpJB2u0000 op_name=retention_delete_check db_instance=ClearAlarms db_shard_group=4141 db_rp=alarm
ts=2020-12-02T02:08:34.616685Z lvl=info msg="Deleted shard group" log_id=0Q6_M9EW000 service=retention trace_id=0QpJB2u0000 op_name=retention_delete_check db_instance=KPIS db_shard_group=4142 db_rp=kpi
ts=2020-12-02T02:08:34.631670Z lvl=info msg="Deleted shard group" log_id=0Q6_M9EW000 service=retention trace_id=0QpJB2u0000 op_name=retention_delete_check db_instance=securityeventlogs db_shard_group=805 db_rp=securityeventlogs
fatal error: concurrent map read and map write

goroutine 495689573 [running]:
runtime.throw(0x1240f52, 0x21)
/usr/local/go/src/runtime/panic.go:608 +0x72 fp=0xc05ecbdb28 sp=0xc05ecbdaf8 pc=0x42c1a2
runtime.mapaccess1_fast64(0x1060e80, 0xc000154e70, 0x1428, 0x0)
/usr/local/go/src/runtime/map_fast64.go:21 +0x195 fp=0xc05ecbdb50 sp=0xc05ecbdb28 pc=0x410b25
github.com/influxdata/influxdb/tsdb.(*Store).WriteToShard(0xc00028b600, 0x1428, 0xc03090e500, 0x1, 0x1, 0x0, 0x0)
/go/src/github.com/influxdata/influxdb/tsdb/store.go:1380 +0x133 fp=0xc05ecbdbc0 sp=0xc05ecbdb50 pc=0x9bfb73
github.com/influxdata/influxdb/coordinator.(*PointsWriter).writeToShard(0xc000291180, 0xc029d242e0, 0xc0435ba01c, 0x9, 0xc000484100, 0x9, 0xc03090e500, 0x1, 0x1, 0x9c9223, ...)
/go/src/github.com/influxdata/influxdb/coordinator/points_writer.go:370 +0x92 fp=0xc05ecbdec8 sp=0xc05ecbdbc0 pc=0xa6bb52
github.com/influxdata/influxdb/coordinator.(*PointsWriter).WritePointsPrivileged.func1(0xc000291180, 0xc012b0d200, 0xc029d242e0, 0xc0435ba01c, 0x9, 0xc000484100, 0x9, 0xc03090e500, 0x1, 0x1)
/go/src/github.com/influxdata/influxdb/coordinator/points_writer.go:312 +0xa8 fp=0xc05ecbdf90 sp=0xc05ecbdec8 pc=0xa7bdd8
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc05ecbdf98 sp=0xc05ecbdf90 pc=0x45b9e1
created by github.com/influxdata/influxdb/coordinator.(*PointsWriter).WritePointsPrivileged
/go/src/github.com/influxdata/influxdb/coordinator/points_writer.go:311 +0x2bb

goroutine 1 [chan receive, 51150 minutes]:
main.(*Main).Run(0xc0003bff58, 0xc0000a8190, 0x0, 0x0, 0xc0003bff68, 0xf4c516)
/go/src/github.com/influxdata/influxdb/cmd/influxd/main.go:90 +0x2d1
main.main()
/go/src/github.com/influxdata/influxdb/cmd/influxd/main.go:45 +0x12f

goroutine 18 [syscall, 51150 minutes]:
os/signal.signal_recv(0x0)
/usr/local/go/src/runtime/sigqueue.go:139 +0x9c
os/signal.loop()
/usr/local/go/src/os/signal/signal_unix.go:23 +0x22
created by os/signal.init.0
/usr/local/go/src/os/signal/signal_unix.go:29 +0x41

goroutine 65 [IO wait, 51150 minutes]:
internal/poll.runtime_pollWait(0x7f9590e93f00, 0x72, 0x0)
/usr/local/go/src/runtime/netpoll.go:173 +0x66
internal/poll.(*pollDesc).wait(0xc000291218, 0x72, 0xc0000c6000, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x9a
internal/poll.(*pollDesc).waitRead(0xc000291218, 0xffffffffffffff00, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Accept(0xc000291200, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:384 +0x1a0
net.(*netFD).accept(0xc000291200, 0x0, 0x0, 0x0)
/usr/local/go/src/net/fd_unix.go:238 +0x42
net.(*TCPListener).accept(0xc000218888, 0x4739ce, 0xc000198840, 0x4738fd)
/usr/local/go/src/net/tcpsock_posix.go:139 +0x2e
net.(*TCPListener).Accept(0xc000218888, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/tcpsock.go:260 +0x47
github.com/influxdata/influxdb/tcp.(*Mux).Serve(0xc000198840, 0x138af00, 0xc000218888, 0xc000218888, 0x0)
/go/src/github.com/influxdata/influxdb/tcp/mux.go:75 +0x92
created by github.com/influxdata/influxdb/cmd/influxd/run.(*Server).Open
/go/src/github.com/influxdata/influxdb/cmd/influxd/run/server.go:387 +0x28b

goroutine 66 [select, 51150 minutes]:
github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/control.(*Controller).run(0xc000314ea0)
/go/src/github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/control/controller.go:290 +0x21f
created by github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/control.New
/go/src/github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/control/controller.go:106 +0x36a

goroutine 67 [select]:
github.com/influxdata/influxdb/tsdb.(*Store).monitorShards(0xc00028b600)
/go/src/github.com/influxdata/influxdb/tsdb/store.go:1895 +0x16f
github.com/influxdata/influxdb/tsdb.(*Store).Open.func1(0xc00028b600)
/go/src/github.com/influxdata/influxdb/tsdb/store.go:231 +0x42
created by github.com/influxdata/influxdb/tsdb.(*Store).Open
/go/src/github.com/influxdata/influxdb/tsdb/store.go:229 +0x2e7

goroutine 68 [select]:
github.com/influxdata/influxdb/services/subscriber.(*Service).run(0xc0003b6600)
/go/src/github.com/influxdata/influxdb/services/subscriber/service.go:239 +0x29c
github.com/influxdata/influxdb/services/subscriber.(*Service).Open.func1(0xc0003b6600)
/go/src/github.com/influxdata/influxdb/services/subscriber/service.go:98 +0x57
created by github.com/influxdata/influxdb/services/subscriber.(*Service).Open
/go/src/github.com/influxdata/influxdb/services/subscriber/service.go:96 +0x160

goroutine 69 [select]:
github.com/influxdata/influxdb/services/subscriber.(*Service).waitForMetaUpdates(0xc0003b6600)
/go/src/github.com/influxdata/influxdb/services/subscriber/service.go:165 +0xcf
github.com/influxdata/influxdb/services/subscriber.(*Service).Open.func2(0xc0003b6600)
/go/src/github.com/influxdata/influxdb/services/subscriber/service.go:102 +0x57
created by github.com/influxdata/influxdb/services/subscriber.(*Service).Open
/go/src/github.com/influxdata/influxdb/services/subscriber/service.go:100 +0x182

goroutine 70 [select]:
github.com/influxdata/influxdb/monitor.(*Monitor).storeStatistics(0xc0004f80f0)
/go/src/github.com/influxdata/influxdb/monitor/service.go:434 +0xb53
created by github.com/influxdata/influxdb/monitor.(*Monitor).Open
/go/src/github.com/influxdata/influxdb/monitor/service.go:126 +0x333

goroutine 71 [select, 9 minutes]:
github.com/influxdata/influxdb/services/precreator.(*Service).runPrecreation(0xc000046300)
/go/src/github.com/influxdata/influxdb/services/precreator/service.go:76 +0x10a
created by github.com/influxdata/influxdb/services/precreator.(*Service).Open
/go/src/github.com/influxdata/influxdb/services/precreator/service.go:54 +0x22b

goroutine 72 [select, 51150 minutes]:
github.com/influxdata/influxdb/tcp.(*listener).Accept(0xc000155980, 0x0, 0x0, 0x0, 0x0)
/go/src/github.com/influxdata/influxdb/tcp/mux.go:236 +0x10b
github.com/influxdata/influxdb/services/snapshotter.(*Service).serve(0xc0003b8780)
/go/src/github.com/influxdata/influxdb/services/snapshotter/service.go:94 +0x7b
created by github.com/influxdata/influxdb/services/snapshotter.(*Service).Open
/go/src/github.com/influxdata/influxdb/services/snapshotter/service.go:68 +0x92

goroutine 73 [select]:
github.com/influxdata/influxdb/services/continuous_querier.(*Service).backgroundLoop(0xc000144e10)
/go/src/github.com/influxdata/influxdb/services/continuous_querier/service.go:215 +0x187
created by github.com/influxdata/influxdb/services/continuous_querier.(*Service).Open
/go/src/github.com/influxdata/influxdb/services/continuous_querier/service.go:133 +0x167

goroutine 74 [IO wait]:
internal/poll.runtime_pollWait(0x7f9590e93e30, 0x72, 0x0)
/usr/local/go/src/runtime/netpoll.go:173 +0x66
internal/poll.(*pollDesc).wait(0xc000291498, 0x72, 0xc00c706e00, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x9a
internal/poll.(*pollDesc).waitRead(0xc000291498, 0xffffffffffffff00, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Accept(0xc000291480, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:384 +0x1a0
net.(*netFD).accept(0xc000291480, 0x83564401, 0x139d97ce686cb42d, 0x44b628)
/usr/local/go/src/net/fd_unix.go:238 +0x42
net.(*TCPListener).accept(0xc000218940, 0x5fc6f71e, 0xc000424df0, 0x48b526)
/usr/local/go/src/net/tcpsock_posix.go:139 +0x2e
net.(*TCPListener).Accept(0xc000218940, 0xc000424e40, 0x18, 0xc000470600, 0x70bc95)
/usr/local/go/src/net/tcpsock.go:260 +0x47
net/http.(*Server).Serve(0xc000204000, 0x138af00, 0xc000218940, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:2826 +0x22f
net/http.Serve(0x138af00, 0xc000218940, 0x1377960, 0xc000293000, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:2423 +0x6e
github.com/influxdata/influxdb/services/httpd.(*Service).serve(0xc000240d20, 0x138af00, 0xc000218940)
/go/src/github.com/influxdata/influxdb/services/httpd/service.go:249 +0x62
github.com/influxdata/influxdb/services/httpd.(*Service).serveTCP(0xc000240d20)
/go/src/github.com/influxdata/influxdb/services/httpd/service.go:237 +0x3c
created by github.com/influxdata/influxdb/services/httpd.(*Service).Open
/go/src/github.com/influxdata/influxdb/services/httpd/service.go:186 +0x6aa

goroutine 75 [runnable]:
github.com/influxdata/influxdb/tsdb.byDatabase.func1(0xc050f8ad80, 0xc004296f00)
/go/src/github.com/influxdata/influxdb/tsdb/store.go:963 +0x66
github.com/influxdata/influxdb/tsdb.(*Store).filterShards(0xc00028b600, 0xc0066b9e20, 0xc0066b9e20, 0xc03f94a0e8, 0x1d6b700)
/go/src/github.com/influxdata/influxdb/tsdb/store.go:953 +0x163
github.com/influxdata/influxdb/tsdb.(*Store).DeleteShard(0xc00028b600, 0x102c, 0x0, 0x0)
/go/src/github.com/influxdata/influxdb/tsdb/store.go:712 +0x1b0
github.com/influxdata/influxdb/services/retention.(*Service).run(0xc0003b8820)
/go/src/github.com/influxdata/influxdb/services/retention/service.go:134 +0x261
github.com/influxdata/influxdb/services/retention.(*Service).Open.func1(0xc0003b8820)
/go/src/github.com/influxdata/influxdb/services/retention/service.go:51 +0x57
created by github.com/influxdata/influxdb/services/retention.(*Service).Open
/go/src/github.com/influxdata/influxdb/services/retention/service.go:51 +0x1ae

goroutine 76 [select, 750 minutes]:
github.com/influxdata/influxdb/cmd/influxd/run.(*Server).startServerReporting(0xc0004e8000)
/go/src/github.com/influxdata/influxdb/cmd/influxd/run/server.go:515 +0x10a
created by github.com/influxdata/influxdb/cmd/influxd/run.(*Server).Open
/go/src/github.com/influxdata/influxdb/cmd/influxd/run/server.go:460 +0xb51

goroutine 77 [select, 51150 minutes]:
github.com/influxdata/influxdb/cmd/influxd/run.(*Command).monitorServerErrors(0xc00020ec60)
/go/src/github.com/influxdata/influxdb/cmd/influxd/run/command.go:174 +0x178
created by github.com/influxdata/influxdb/cmd/influxd/run.(*Command).Run
/go/src/github.com/influxdata/influxdb/cmd/influxd/run/command.go:155 +0xefd

goroutine 492849952 [select]:
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).compact(0xc02712d320, 0xc0066fd3d0)
/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1967 +0x25b
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableLevelCompactions.func1(0xc0066fd3d0, 0xc02712d320)
/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:420 +0x5b
created by github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableLevelCompactions
/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:420 +0x132

goroutine 495571728 [select]:
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).compactCache(0xc0544b5b00)
/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1920 +0x11b
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableSnapshotCompactions.func1(0xc003f41850, 0xc0544b5b00)
/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:499 +0x51
created by github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableSnapshotCompactions
/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:499 +0x124

Start with tsdb/store.go at WriteToShard around line 1380 and trace the concurrent shard access alongside DeleteShard and filterShards, which appear in the stack trace. Review the retention service path in services/retention/service.go and determine how the crash can be reproduced. Done means the concurrent map panic is prevented and the relevant behavior is covered by tests.

Written by the indexing model from the issue text.