goroutine leak when call func CreateSeriesListIfNotExists on v1.8.0

__Steps to reproduce:__
1. Just running the influxd program

__Expected behavior:__
Always running normally.

__Actual behavior:__
At the begin time, the influxd running as normal to receive http request & do some compactions, but after 4-5 days. the running influxd reject all http request and response 500. the machine memory consumed about 59g on the same time and seems growing slowly.
The situation lasting about 3 hours util i restarted the program. but what mad me is that it ocurred again. i got goroutines info by gopprof tools to find what happened and there was total 79733 goroutines.

__Environment info:__
1.Machine info:
physical machine, cpu: 32core, RAM:64G, HDD: 2T

2. I doubt the HDD maybe exist problems and had done some tests:
```
#write testing:
> time dd if=/dev/zero bs=1024 count=1000000 of=/data/1Gb.file
1000000+0 records in
1000000+0 records out
1024000000 bytes (1.0 GB) copied, 2.74689 s, 373 MB/s

real 0m2.799s
user 0m0.309s
sys 0m2.437s

# read testing:
> hdparm -tT /dev/sda
/dev/sda:
Timing cached reads: 17918 MB in 1.99 seconds = 8991.41 MB/sec
Timing buffered disk reads: 992 MB in 3.01 seconds = 330.07 MB/sec
```

3. We use http-batch method write data to influxdb, about 200MBbytes/min, and current retention policy is **duration 3d REPLICATION 1 SHARD DURATION 1d**

* System info: Run `uname -srm` and copy the output here
CentOS Linux release 7.4.1708 (Core)
Linux 3.10.0-957.27.2.el7.x86_64 x86_64

* InfluxDB version: Run `influxd version` and copy the output here
InfluxDB v1.8.0 (git: 1.8 781490de48220d7695a05c29e5a36f550a4568f5)

* Other relevant environment details: Container runtime, disk info, etc
we use physical machine

__Config:__
```toml
[data]
wal-fsync-delay = "50ms"

index-version = "tsi1"

cache-max-memory-size = "2g"

cache-snapshot-memory-size = "128m"

cache-snapshot-write-cold-duration = "30m"

#the retention policy duration is 72h
#we setted 80h meaning not execute full compactions
compact-full-write-cold-duration = "80h"

max-concurrent-compactions = 8

compact-throughput = "16m"

compact-throughput-burst = "16m"

max-index-log-file-size = "128m"

[monitor]
store-enabled = false
```

__Logs:__
i have commented some sensitive data:
```
[httpd] - [13/Jan/2021:11:18:26 +0800] "POST /write?* HTTP/1.1" 500 20 "-" "Go-http-client/1.1" 0031c41b-554e-11eb-8ad5-c81fbea4cf8a 63144337
[httpd] - [13/Jan/2021:11:18:26 +0800] "POST /write?* HTTP/1.1" 500 20 "-" "Go-http-client/1.1" 0031931f-554e-11eb-8ad1-c81fbea4cf8a 63151839
[httpd] - [13/Jan/2021:11:18:26 +0800] "POST /write?* HTTP/1.1" 500 20 "-" "Go-http-client/1.1" 0037ba9e-554e-11eb-8ae2-c81fbea4cf8a 53913130
[httpd] - [13/Jan/2021:11:18:36 +0800] "POST /write?* HTTP/1.1" 500 20 "-" "Go-http-client/1.1" 0604feb8-554e-11eb-8afa-c81fbea4cf8a 55115085
[httpd] - [13/Jan/2021:11:18:26 +0800] "POST /write?* HTTP/1.1" 500 20 "-" "Go-http-client/1.1" 002f9f19-554e-11eb-8ac7-c81fbea4cf8a 69153003
ts=2021-01-13T03:19:29.100453Z lvl=error msg="[500] - \"timeout\"" log_id=0R_GfS8G000 service=httpd
ts=2021-01-13T03:19:28.988371Z lvl=error msg="[500] - \"timeout\"" log_id=0R_GfS8G000 service=httpd
ts=2021-01-13T03:19:29.028376Z lvl=error msg="[500] - \"timeout\"" log_id=0R_GfS8G000 service=httpd
ts=2021-01-13T03:19:39.233851Z lvl=error msg="[500] - \"timeout\"" log_id=0R_GfS8G000 service=httpd
ts=2021-01-13T03:19:39.304027Z lvl=error msg="[500] - \"timeout\"" log_id=0R_GfS8G000 service=httpd
ts=2021-01-13T03:19:29.029345Z lvl=error msg="[500] - \"timeout\"" log_id=0R_GfS8G000 service=httpd
```

__Performance:__
attached pprof goroutine infos:
```
goroutine profile: total 79733
46490 @ 0x432fe0 0x443650 0x44363b 0x4433b7 0x484d9c 0x486467 0x4863f2 0xa0f6eb 0xa330ef 0x98f9c4 0x460f61
# 0x4433b6 sync.runtime_SemacquireMutex+0x46 /usr/local/go/src/runtime/sema.go:71
# 0x484d9b sync.(*Mutex).lockSlow+0xfb /usr/local/go/src/sync/mutex.go:138
# 0x486466 sync.(*Mutex).Lock+0x96 /usr/local/go/src/sync/mutex.go:81
# 0x4863f1 sync.(*RWMutex).Lock+0x21 /usr/local/go/src/sync/rwmutex.go:98
# 0xa0f6ea github.com/influxdata/influxdb/tsdb.(*SeriesPartition).CreateSeriesListIfNotExists+0x1ea /go/src/github.com/influxdata/influxdb/tsdb/series_partition.go:231
# 0xa330ee github.com/influxdata/influxdb/tsdb.(*SeriesFile).CreateSeriesListIfNotExists.func1+0x7e /go/src/github.com/influxdata/influxdb/tsdb/series_file.go:187
# 0x98f9c3 github.com/influxdata/influxdb/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1+0x63 /go/src/github.com/influxdata/influxdb/vendor/golang.org/x/sync/errgroup/errgroup.go:57

24828 @ 0x432fe0 0x443650 0x44363b 0x4432a2 0x486844 0x98f831 0xa09d3e 0x11bb537 0x11c98a8 0x11d6217 0x460f61
# 0x4432a1 sync.runtime_Semacquire+0x41 /usr/local/go/src/runtime/sema.go:56
# 0x486843 sync.(*WaitGroup).Wait+0x63 /usr/local/go/src/sync/waitgroup.go:130
# 0x98f830 github.com/influxdata/influxdb/vendor/golang.org/x/sync/errgroup.(*Group).Wait+0x30 /go/src/github.com/influxdata/influxdb/vendor/golang.org/x/sync/errgroup/errgroup.go:40
# 0xa09d3d github.com/influxdata/influxdb/tsdb.(*SeriesFile).CreateSeriesListIfNotExists+0x2cd /go/src/github.com/influxdata/influxdb/tsdb/series_file.go:190
# 0x11bb536 github.com/influxdata/influxdb/tsdb/index/tsi1.(*LogFile).AddSeriesList+0xb6 /go/src/github.com/influxdata/influxdb/tsdb/index/tsi1/log_file.go:515
# 0x11c98a7 github.com/influxdata/influxdb/tsdb/index/tsi1.(*Partition).createSeriesListIfNotExists+0x157 /go/src/github.com/influxdata/influxdb/tsdb/index/tsi1/partition.go:679
# 0x11d6216 github.com/influxdata/influxdb/tsdb/index/tsi1.(*Index).CreateSeriesListIfNotExists.func1+0x416 /go/src/github.com/influxdata/influxdb/tsdb/index/tsi1/index.go:667

6195 @ 0x432fe0 0x4077f8 0x4077ce 0x4074bb 0x11afbcb 0x12250ca 0xa1884d 0xa18229 0xa2dae8 0xae0730 0xaf17d8 0x460f61
# 0x11afbca github.com/influxdata/influxdb/tsdb/index/tsi1.(*Index).CreateSeriesListIfNotExists+0x40a /go/src/github.com/influxdata/influxdb/tsdb/index/tsi1/index.go:720
# 0x12250c9 github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).CreateSeriesListIfNotExists+0xa9 /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1835
# 0xa1884c github.com/influxdata/influxdb/tsdb.(*Shard).validateSeriesAndFields+0x30c /go/src/github.com/influxdata/influxdb/tsdb/shard.go:592
# 0xa18228 github.com/influxdata/influxdb/tsdb.(*Shard).WritePoints+0x128 /go/src/github.com/influxdata/influxdb/tsdb/shard.go:508
# 0xa2dae7 github.com/influxdata/influxdb/tsdb.(*Store).WriteToShard+0x247 /go/src/github.com/influxdata/influxdb/tsdb/store.go:1411
# 0xae072f github.com/influxdata/influxdb/coordinator.(*PointsWriter).writeToShard+0x8f /go/src/github.com/influxdata/influxdb/coordinator/points_writer.go:370
# 0xaf17d7 github.com/influxdata/influxdb/coordinator.(*PointsWriter).WritePointsPrivileged.func1+0xa7 /go/src/github.com/influxdata/influxdb/coordinator/points_writer.go:312

1655 @ 0x432fe0 0x443650 0x44363b 0x4433b7 0xa100b6 0xa0f53e 0xa330ef 0x98f9c4 0x460f61
# 0x4433b6 sync.runtime_SemacquireMutex+0x46 /usr/local/go/src/runtime/sema.go:71
# 0xa100b5 sync.(*RWMutex).RLock+0xbb5 /usr/local/go/src/sync/rwmutex.go:50
# 0xa0f53d github.com/influxdata/influxdb/tsdb.(*SeriesPartition).CreateSeriesListIfNotExists+0x3d /go/src/github.com/influxdata/influxdb/tsdb/series_partition.go:201
# 0xa330ee github.com/influxdata/influxdb/tsdb.(*SeriesFile).CreateSeriesListIfNotExists.func1+0x7e /go/src/github.com/influxdata/influxdb/tsdb/series_file.go:187
# 0x98f9c3 github.com/influxdata/influxdb/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1+0x63 /go/src/github.com/influxdata/influxdb/vendor/golang.org/x/sync/errgroup/errgroup.go:57

230 @ 0x9910d4 0xa0cfb9 0xa0f5fe 0xa330ef 0x98f9c4 0x460f61
# 0x9910d3 github.com/influxdata/influxdb/pkg/rhh.HashKey+0x73 /go/src/github.com/influxdata/influxdb/pkg/rhh/rhh.go:237
# 0xa0cfb8 github.com/influxdata/influxdb/tsdb.(*SeriesIndex).FindIDBySeriesKey+0x208 /go/src/github.com/influxdata/influxdb/tsdb/series_index.go:205
# 0xa0f5fd github.com/influxdata/influxdb/tsdb.(*SeriesPartition).CreateSeriesListIfNotExists+0xfd /go/src/github.com/influxdata/influxdb/tsdb/series_partition.go:210
# 0xa330ee github.com/influxdata/influxdb/tsdb.(*SeriesFile).CreateSeriesListIfNotExists.func1+0x7e /go/src/github.com/influxdata/influxdb/tsdb/series_file.go:187
# 0x98f9c3 github.com/influxdata/influxdb/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1+0x63 /go/src/github.com/influxdata/influxdb/vendor/golang.org/x/sync/errgroup/errgroup.go:57

76 @ 0xa0d296 0xa0f5fe 0xa330ef 0x98f9c4 0x460f61
# 0xa0d295 github.com/influxdata/influxdb/tsdb.(*SeriesIndex).FindIDBySeriesKey+0x4e5 /go/src/github.com/influxdata/influxdb/tsdb/series_index.go:185
# 0xa0f5fd github.com/influxdata/influxdb/tsdb.(*SeriesPartition).CreateSeriesListIfNotExists+0xfd /go/src/github.com/influxdata/influxdb/tsdb/series_partition.go:210
# 0xa330ee github.com/influxdata/influxdb/tsdb.(*SeriesFile).CreateSeriesListIfNotExists.func1+0x7e /go/src/github.com/influxdata/influxdb/tsdb/series_file.go:187
# 0x98f9c3 github.com/influxdata/influxdb/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1+0x63 /go/src/github.com/influxdata/influxdb/vendor/golang.org/x/sync/errgroup/errgroup.go:57

72 @ 0x432fe0 0x42dd7a 0x42d345 0x4d3815 0x4d478f 0x4d4771 0x63b49f 0x64ef18 0x762a88 0x460f61
# 0x42d344 internal/poll.runtime_pollWait+0x54 /usr/local/go/src/runtime/netpoll.go:184
# 0x4d3814 internal/poll.(*pollDesc).wait+0x44 /usr/local/go/src/internal/poll/fd_poll_runtime.go:87
# 0x4d478e internal/poll.(*pollDesc).waitRead+0x1ce /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
# 0x4d4770 internal/poll.(*FD).Read+0x1b0 /usr/local/go/src/internal/poll/fd_unix.go:169
# 0x63b49e net.(*netFD).Read+0x4e /usr/local/go/src/net/fd_unix.go:202
# 0x64ef17 net.(*conn).Read+0x67 /usr/local/go/src/net/net.go:184
# 0x762a87 net/http.(*connReader).backgroundRead+0x57 /usr/local/go/src/net/http/server.go:677

71 @ 0x432fe0 0x442aeb 0xae0311 0xadfc71 0xf4cb07 0xf4be7f 0xf61fb1 0x769b04 0xf62efb 0x769b04 0xf60625 0x769b04 0xf620ae 0x769b04 0xf629dd 0x769b04 0xf62b7f 0x769b04 0xf6343b 0x769b04 0xd631c3 0xf48a05 0x76cf54 0x7688f5 0x460f61
# 0xae0310 github.com/influxdata/influxdb/coordinator.(*PointsWriter).WritePointsPrivileged+0x670 /go/src/github.com/influxdata/influxdb/coordinator/points_writer.go:350
# 0xadfc70 github.com/influxdata/influxdb/coordinator.(*PointsWriter).WritePoints+0x90 /go/src/github.com/influxdata/influxdb/coordinator/points_writer.go:287
# 0xf4cb06 github.com/influxdata/influxdb/services/httpd.(*Handler).serveWrite+0x946 /go/src/github.com/influxdata/influxdb/services/httpd/handler.go:962
# 0xf4be7e github.com/influxdata/influxdb/services/httpd.(*Handler).serveWriteV1+0x14e /go/src/github.com/influxdata/influxdb/services/httpd/handler.go:854
# 0xf61fb0 github.com/influxdata/influxdb/services/httpd.authenticate.func1+0x7e0 /go/src/github.com/influxdata/influxdb/services/httpd/handler.go:1732
# 0x769b03 net/http.HandlerFunc.ServeHTTP+0x43 /usr/local/go/src/net/http/server.go:2007
# 0xf62efa github.com/influxdata/influxdb/services/httpd.(*Handler).responseWriter.func1+0xaa /go/src/github.com/influxdata/influxdb/services/httpd/handler.go:1917
# 0x769b03 net/http.HandlerFunc.ServeHTTP+0x43 /usr/local/go/src/net/http/server.go:2007
# 0xf60624 github.com/influxdata/influxdb/services/httpd.gzipFilter.func1+0x244 /go/src/github.com/influxdata/influxdb/services/httpd/gzip.go:39
# 0x769b03 net/http.HandlerFunc.ServeHTTP+0x43 /usr/local/go/src/net/http/server.go:2007
# 0xf620ad github.com/influxdata/influxdb/services/httpd.cors.func1+0xdd /go/src/github.com/influxdata/influxdb/services/httpd/handler.go:1859
# 0x769b03 net/http.HandlerFunc.ServeHTTP+0x43 /usr/local/go/src/net/http/server.go:2007
# 0xf629dc github.com/influxdata/influxdb/services/httpd.requestID.func1+0x34c /go/src/github.com/influxdata/influxdb/services/httpd/handler.go:1890
# 0x769b03 net/http.HandlerFunc.ServeHTTP+0x43 /usr/local/go/src/net/http/server.go:2007
# 0xf62b7e github.com/influxdata/influxdb/services/httpd.(*Handler).logging.func1+0xde /go/src/github.com/influxdata/influxdb/services/httpd/handler.go:1898
# 0x769b03 net/http.HandlerFunc.ServeHTTP+0x43 /usr/local/go/src/net/http/server.go:2007
# 0xf6343a github.com/influxdata/influxdb/services/httpd.(*Handler).recovery.func1+0x17a /go/src/github.com/influxdata/influxdb/services/httpd/handler.go:1954
# 0x769b03 net/http.HandlerFunc.ServeHTTP+0x43 /usr/local/go/src/net/http/server.go:2007
# 0xd631c2 github.com/influxdata/influxdb/vendor/github.com/bmizerany/pat.(*PatternServeMux).ServeHTTP+0x152 /go/src/github.com/influxdata/influxdb/vendor/github.com/bmizerany/pat/mux.go:117
# 0xf48a04 github.com/influxdata/influxdb/services/httpd.(*Handler).ServeHTTP+0x394 /go/src/github.com/influxdata/influxdb/services/httpd/handler.go:454
# 0x76cf53 net/http.serverHandler.ServeHTTP+0xa3 /usr/local/go/src/net/http/server.go:2802
# 0x7688f4 net/http.(*conn).serve+0x874 /usr/local/go/src/net/http/server.go:1890

63 @ 0xa139fd 0xa0cf8a 0xa0f5fe 0xa330ef 0x98f9c4 0x460f61
# 0xa139fc github.com/influxdata/influxdb/tsdb.ReadSeriesKeyFromSegments+0x12c /go/src/github.com/influxdata/influxdb/tsdb/series_segment.go:330
# 0xa0cf89 github.com/influxdata/influxdb/tsdb.(*SeriesIndex).FindIDBySeriesKey+0x1d9 /go/src/github.com/influxdata/influxdb/tsdb/series_index.go:204
# 0xa0f5fd github.com/influxdata/influxdb/tsdb.(*SeriesPartition).CreateSeriesListIfNotExists+0xfd /go/src/github.com/influxdata/influxdb/tsdb/series_partition.go:210
# 0xa330ee github.com/influxdata/influxdb/tsdb.(*SeriesFile).CreateSeriesListIfNotExists.func1+0x7e /go/src/github.com/influxdata/influxdb/tsdb/series_file.go:187
# 0x98f9c3 github.com/influxdata/influxdb/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1+0x63 /go/src/github.com/influxdata/influxdb/vendor/golang.org/x/sync/errgroup/errgroup.go:57

8 @ 0x432fe0 0x443650 0x44363b 0x4433b7 0x486458 0xa0f6eb 0xa330ef 0x98f9c4 0x460f61
# 0x4433b6 sync.runtime_SemacquireMutex+0x46 /usr/local/go/src/runtime/sema.go:71
# 0x486457 sync.(*RWMutex).Lock+0x87 /usr/local/go/src/sync/rwmutex.go:103
# 0xa0f6ea github.com/influxdata/influxdb/tsdb.(*SeriesPartition).CreateSeriesListIfNotExists+0x1ea /go/src/github.com/influxdata/influxdb/tsdb/series_partition.go:231
# 0xa330ee github.com/influxdata/influxdb/tsdb.(*SeriesFile).CreateSeriesListIfNotExists.func1+0x7e /go/src/github.com/influxdata/influxdb/tsdb/series_file.go:187
# 0x98f9c3 github.com/influxdata/influxdb/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1+0x63 /go/src/github.com/influxdata/influxdb/vendor/golang.org/x/sync/errgroup/errgroup.go:57

8 @ 0x4863c9 0xa1001a 0xa0f693 0xa330ef 0x98f9c4 0x460f61
# 0x4863c8 sync.(*RWMutex).rUnlockSlow+0x88 /usr/local/go/src/sync/rwmutex.go:77
# 0xa10019 sync.(*RWMutex).RUnlock+0xb19 /usr/local/go/src/sync/rwmutex.go:70
# 0xa0f692 github.com/influxdata/influxdb/tsdb.(*SeriesPartition).CreateSeriesListIfNotExists+0x192 /go/src/github.com/influxdata/influxdb/tsdb/series_partition.go:217
# 0xa330ee github.com/influxdata/influxdb/tsdb.(*SeriesFile).CreateSeriesListIfNotExists.func1+0x7e /go/src/github.com/influxdata/influxdb/tsdb/series_file.go:187
# 0x98f9c3 github.com/influxdata/influxdb/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1+0x63 /go/src/github.com/influxdata/influxdb/vendor/golang.org/x/sync/errgroup/errgroup.go:57

7 @ 0x41344b 0xa0d084 0xa0d091 0xa0f5fe 0xa330ef 0x98f9c4 0x460f61
# 0xa0d083 github.com/influxdata/influxdb/tsdb.(*SeriesIndex).IsDeleted+0x2d3 /go/src/github.com/influxdata/influxdb/tsdb/series_index.go:158
# 0xa0d090 github.com/influxdata/influxdb/tsdb.(*SeriesIndex).FindIDBySeriesKey+0x2e0 /go/src/github.com/influxdata/influxdb/tsdb/series_index.go:210
# 0xa0f5fd github.com/influxdata/influxdb/tsdb.(*SeriesPartition).CreateSeriesListIfNotExists+0xfd /go/src/github.com/influxdata/influxdb/tsdb/series_partition.go:210
# 0xa330ee github.com/influxdata/influxdb/tsdb.(*SeriesFile).CreateSeriesListIfNotExists.func1+0x7e /go/src/github.com/influxdata/influxdb/tsdb/series_file.go:187
# 0x98f9c3 github.com/influxdata/influxdb/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1+0x63 /go/src/github.com/influxdata/influxdb/vendor/golang.org/x/sync/errgroup/errgroup.go:57

5 @ 0x432fe0 0x442aeb 0x12266a2 0x1272ebb 0x460f61
# 0x12266a1 github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).compactCache+0x131 /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1976
# 0x1272eba github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableSnapshotCompactions.func1+0x5a /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:505

5 @ 0x432fe0 0x442aeb 0x122703f 0x1272e35 0x460f61
# 0x122703e github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).compact+0x26e /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:2023
# 0x1272e34 github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableLevelCompactions.func1+0x64 /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:426

3 @ 0x99114d 0xa0d74a 0xa0d0e8 0xa0d091 0xa0f5fe 0xa330ef 0x98f9c4 0x460f61
# 0x99114c github.com/influxdata/influxdb/pkg/rhh.HashUint64+0x6c /go/src/github.com/influxdata/influxdb/pkg/rhh/rhh.go:248
# 0xa0d749 github.com/influxdata/influxdb/tsdb.(*SeriesIndex).FindOffsetByID+0x119 /go/src/github.com/influxdata/influxdb/tsdb/series_index.go:253
# 0xa0d0e7 github.com/influxdata/influxdb/tsdb.(*SeriesIndex).IsDeleted+0x337 /go/src/github.com/influxdata/influxdb/tsdb/series_index.go:161
# 0xa0d090 github.com/influxdata/influxdb/tsdb.(*SeriesIndex).FindIDBySeriesKey+0x2e0 /go/src/github.com/influxdata/influxdb/tsdb/series_index.go:210
# 0xa0f5fd github.com/influxdata/influxdb/tsdb.(*SeriesPartition).CreateSeriesListIfNotExists+0xfd /go/src/github.com/influxdata/influxdb/tsdb/series_partition.go:210
# 0xa330ee github.com/influxdata/influxdb/tsdb.(*SeriesFile).CreateSeriesListIfNotExists.func1+0x7e /go/src/github.com/influxdata/influxdb/tsdb/series_file.go:187
# 0x98f9c3 github.com/influxdata/influxdb/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1+0x63 /go/src/github.com/influxdata/influxdb/vendor/golang.org/x/sync/errgroup/errgroup.go:57

1 @ 0x40cc64 0x44768c 0x5080f2 0x460f61
# 0x44768b os/signal.signal_recv+0x9b /usr/local/go/src/runtime/sigqueue.go:147
# 0x5080f1 os/signal.loop+0x21 /usr/local/go/src/os/signal/signal_unix.go:23

1 @ 0x432fe0 0x4077f8 0x4077ce 0x4074bb 0x12925e9 0x129227d 0x432c0e 0x460f61
# 0x12925e8 main.(*Main).Run+0x2c8 /go/src/github.com/influxdata/influxdb/cmd/influxd/main.go:90
# 0x129227c main.main+0x13c /go/src/github.com/influxdata/influxdb/cmd/influxd/main.go:45
# 0x432c0d runtime.main+0x21d /usr/local/go/src/runtime/proc.go:203

1 @ 0x432fe0 0x42dd7a 0x42d345 0x4d3815 0x4d478f 0x4d4771 0x63b49f 0x64ef18 0x762ed4 0x5364c3 0x53662f 0x764bab 0x768754 0x460f61
# 0x42d344 internal/poll.runtime_pollWait+0x54 /usr/local/go/src/runtime/netpoll.go:184
# 0x4d3814 internal/poll.(*pollDesc).wait+0x44 /usr/local/go/src/internal/poll/fd_poll_runtime.go:87
# 0x4d478e internal/poll.(*pollDesc).waitRead+0x1ce /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
# 0x4d4770 internal/poll.(*FD).Read+0x1b0 /usr/local/go/src/internal/poll/fd_unix.go:169
# 0x63b49e net.(*netFD).Read+0x4e /usr/local/go/src/net/fd_unix.go:202
# 0x64ef17 net.(*conn).Read+0x67 /usr/local/go/src/net/net.go:184
# 0x762ed3 net/http.(*connReader).Read+0xf3 /usr/local/go/src/net/http/server.go:785
# 0x5364c2 bufio.(*Reader).fill+0x102 /usr/local/go/src/bufio/bufio.go:100
# 0x53662e bufio.(*Reader).Peek+0x4e /usr/local/go/src/bufio/bufio.go:138
# 0x764baa net/http.(*conn).readRequest+0xb3a /usr/local/go/src/net/http/server.go:962
# 0x768753 net/http.(*conn).serve+0x6d3 /usr/local/go/src/net/http/server.go:1817

1 @ 0x432fe0 0x42dd7a 0x42d345 0x4d3815 0x4d478f 0x4d4771 0x63b49f 0x64ef18 0x762ed4 0x5364c3 0x53721d 0x537454 0x6ea26c 0x75d4b2 0x75d4e1 0x7641cf 0x768754 0x460f61
# 0x42d344 internal/poll.runtime_pollWait+0x54 /usr/local/go/src/runtime/netpoll.go:184
# 0x4d3814 internal/poll.(*pollDesc).wait+0x44 /usr/local/go/src/internal/poll/fd_poll_runtime.go:87
# 0x4d478e internal/poll.(*pollDesc).waitRead+0x1ce /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
# 0x4d4770 internal/poll.(*FD).Read+0x1b0 /usr/local/go/src/internal/poll/fd_unix.go:169
# 0x63b49e net.(*netFD).Read+0x4e /usr/local/go/src/net/fd_unix.go:202
# 0x64ef17 net.(*conn).Read+0x67 /usr/local/go/src/net/net.go:184
# 0x762ed3 net/http.(*connReader).Read+0xf3 /usr/local/go/src/net/http/server.go:785
# 0x5364c2 bufio.(*Reader).fill+0x102 /usr/local/go/src/bufio/bufio.go:100
# 0x53721c bufio.(*Reader).ReadSlice+0x3c /usr/local/go/src/bufio/bufio.go:359
# 0x537453 bufio.(*Reader).ReadLine+0x33 /usr/local/go/src/bufio/bufio.go:388
# 0x6ea26b net/textproto.(*Reader).readLineSlice+0x6b /usr/local/go/src/net/textproto/reader.go:57
# 0x75d4b1 net/textproto.(*Reader).ReadLine+0x91 /usr/local/go/src/net/textproto/reader.go:38
# 0x75d4e0 net/http.readRequest+0xc0 /usr/local/go/src/net/http/request.go:1012
# 0x7641ce net/http.(*conn).readRequest+0x15e /usr/local/go/src/net/http/server.go:965
# 0x768753 net/http.(*conn).serve+0x6d3 /usr/local/go/src/net/http/server.go:1817

1 @ 0x432fe0 0x42dd7a 0x42d345 0x4d3815 0x4d478f 0x4d4771 0x63b49f 0x64ef18 0x762ed4 0x536b8a 0x4b0923 0x778edf 0x778e12 0x4b0923 0xf56720 0x511624 0xf4c52e 0xf4be7f 0xf61fb1 0x769b04 0xf62efb 0x769b04 0xf60625 0x769b04 0xf620ae 0x769b04 0xf629dd 0x769b04 0xf62b7f 0x769b04 0xf6343b 0x769b04
# 0x42d344 internal/poll.runtime_pollWait+0x54 /usr/local/go/src/runtime/netpoll.go:184
# 0x4d3814 internal/poll.(*pollDesc).wait+0x44 /usr/local/go/src/internal/poll/fd_poll_runtime.go:87
# 0x4d478e internal/poll.(*pollDesc).waitRead+0x1ce /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
# 0x4d4770 internal/poll.(*FD).Read+0x1b0 /usr/local/go/src/internal/poll/fd_unix.go:169
# 0x63b49e net.(*netFD).Read+0x4e /usr/local/go/src/net/fd_unix.go:202
# 0x64ef17 net.(*conn).Read+0x67 /usr/local/go/src/net/net.go:184
# 0x762ed3 net/http.(*connReader).Read+0xf3 /usr/local/go/src/net/http/server.go:785
# 0x536b89 bufio.(*Reader).Read+0x269 /usr/local/go/src/bufio/bufio.go:226
# 0x4b0922 io.(*LimitedReader).Read+0x62 /usr/local/go/src/io/io.go:448
# 0x778ede net/http.(*body).readLocked+0x5e /usr/local/go/src/net/http/transfer.go:847
# 0x778e11 net/http.(*body).Read+0x101 /usr/local/go/src/net/http/transfer.go:839
# 0x4b0922 io.(*LimitedReader).Read+0x62 /usr/local/go/src/io/io.go:448
# 0xf5671f github.com/influxdata/influxdb/services/httpd.(*truncatedReader).Read+0x4f /go/src/github.com/influxdata/influxdb/services/httpd/io.go:31
# 0x511623 bytes.(*Buffer).ReadFrom+0xb3 /usr/local/go/src/bytes/buffer.go:204
# 0xf4c52d github.com/influxdata/influxdb/services/httpd.(*Handler).serveWrite+0x36d /go/src/github.com/influxdata/influxdb/services/httpd/handler.go:919
# 0xf4be7e github.com/influxdata/influxdb/services/httpd.(*Handler).serveWriteV1+0x14e /go/src/github.com/influxdata/influxdb/services/httpd/handler.go:854
# 0xf61fb0 github.com/influxdata/influxdb/services/httpd.authenticate.func1+0x7e0 /go/src/github.com/influxdata/influxdb/services/httpd/handler.go:1732
# 0x769b03 net/http.HandlerFunc.ServeHTTP+0x43 /usr/local/go/src/net/http/server.go:2007
# 0xf62efa github.com/influxdata/influxdb/services/httpd.(*Handler).responseWriter.func1+0xaa /go/src/github.com/influxdata/influxdb/services/httpd/handler.go:1917
# 0x769b03 net/http.HandlerFunc.ServeHTTP+0x43 /usr/local/go/src/net/http/server.go:2007
# 0xf60624 github.com/influxdata/influxdb/services/httpd.gzipFilter.func1+0x244 /go/src/github.com/influxdata/influxdb/services/httpd/gzip.go:39
# 0x769b03 net/http.HandlerFunc.ServeHTTP+0x43 /usr/local/go/src/net/http/server.go:2007
# 0xf620ad github.com/influxdata/influxdb/services/httpd.cors.func1+0xdd /go/src/github.com/influxdata/influxdb/services/httpd/handler.go:1859
# 0x769b03 net/http.HandlerFunc.ServeHTTP+0x43 /usr/local/go/src/net/http/server.go:2007
# 0xf629dc github.com/influxdata/influxdb/services/httpd.requestID.func1+0x34c /go/src/github.com/influxdata/influxdb/services/httpd/handler.go:1890
# 0x769b03 net/http.HandlerFunc.ServeHTTP+0x43 /usr/local/go/src/net/http/server.go:2007
# 0xf62b7e github.com/influxdata/influxdb/services/httpd.(*Handler).logging.func1+0xde /go/src/github.com/influxdata/influxdb/services/httpd/handler.go:1898
# 0x769b03 net/http.HandlerFunc.ServeHTTP+0x43 /usr/local/go/src/net/http/server.go:2007
# 0xf6343a github.com/influxdata/influxdb/services/httpd.(*Handler).recovery.func1+0x17a /go/src/github.com/influxdata/influxdb/services/httpd/handler.go:1954
# 0x769b03 net/http.HandlerFunc.ServeHTTP+0x43 /usr/local/go/src/net/http/server.go:2007

1 @ 0x432fe0 0x42dd7a 0x42d345 0x4d3815 0x4d6198 0x4d6177 0x63be12 0x659442 0x657fc7 0x76d310 0xf6008c 0xf60020 0xf5ff7c 0x460f61
# 0x42d344 internal/poll.runtime_pollWait+0x54 /usr/local/go/src/runtime/netpoll.go:184
# 0x4d3814 internal/poll.(*pollDesc).wait+0x44 /usr/local/go/src/internal/poll/fd_poll_runtime.go:87
# 0x4d6197 internal/poll.(*pollDesc).waitRead+0x1f7 /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
# 0x4d6176 internal/poll.(*FD).Accept+0x1d6 /usr/local/go/src/internal/poll/fd_unix.go:384
# 0x63be11 net.(*netFD).accept+0x41 /usr/local/go/src/net/fd_unix.go:238
# 0x659441 net.(*TCPListener).accept+0x31 /usr/local/go/src/net/tcpsock_posix.go:139
# 0x657fc6 net.(*TCPListener).Accept+0x46 /usr/local/go/src/net/tcpsock.go:261
# 0x76d30f net/http.(*Server).Serve+0x27f /usr/local/go/src/net/http/server.go:2896
# 0xf6008b net/http.Serve+0x9b /usr/local/go/src/net/http/server.go:2468
# 0xf6001f github.com/influxdata/influxdb/services/httpd.(*Service).serve+0x2f /go/src/github.com/influxdata/influxdb/services/httpd/service.go:249
# 0xf5ff7b github.com/influxdata/influxdb/services/httpd.(*Service).serveTCP+0x3b /go/src/github.com/influxdata/influxdb/services/httpd/service.go:237

1 @ 0x432fe0 0x42dd7a 0x42d345 0x4d3815 0x4d6198 0x4d6177 0x63be12 0x659442 0x657fc7 0x97f8e2 0x460f61
# 0x42d344 internal/poll.runtime_pollWait+0x54 /usr/local/go/src/runtime/netpoll.go:184
# 0x4d3814 internal/poll.(*pollDesc).wait+0x44 /usr/local/go/src/internal/poll/fd_poll_runtime.go:87
# 0x4d6197 internal/poll.(*pollDesc).waitRead+0x1f7 /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
# 0x4d6176 internal/poll.(*FD).Accept+0x1d6 /usr/local/go/src/internal/poll/fd_unix.go:384
# 0x63be11 net.(*netFD).accept+0x41 /usr/local/go/src/net/fd_unix.go:238
# 0x659441 net.(*TCPListener).accept+0x31 /usr/local/go/src/net/tcpsock_posix.go:139
# 0x657fc6 net.(*TCPListener).Accept+0x46 /usr/local/go/src/net/tcpsock.go:261
# 0x97f8e1 github.com/influxdata/influxdb/tcp.(*Mux).Serve+0x91 /go/src/github.com/influxdata/influxdb/tcp/mux.go:75

1 @ 0x432fe0 0x442aeb 0x1287b38 0x460f61
# 0x1287b37 github.com/influxdata/influxdb/cmd/influxd/run.(*Command).monitorServerErrors+0x177 /go/src/github.com/influxdata/influxdb/cmd/influxd/run/command.go:174

1 @ 0x432fe0 0x442aeb 0x98097d 0xa41cd5 0x460f61
# 0x98097c github.com/influxdata/influxdb/tcp.(*listener).Accept+0x11c /go/src/github.com/influxdata/influxdb/tcp/mux.go:236
# 0xa41cd4 github.com/influxdata/influxdb/services/snapshotter.(*Service).serve+0x84 /go/src/github.com/influxdata/influxdb/services/snapshotter/service.go:94

1 @ 0x432fe0 0x442aeb 0xa317ee 0xa3574b 0x460f61
# 0xa317ed github.com/influxdata/influxdb/tsdb.(*Store).monitorShards+0x19d /go/src/github.com/influxdata/influxdb/tsdb/store.go:1911
# 0xa3574a github.com/influxdata/influxdb/tsdb.(*Store).Open.func1+0x4a /go/src/github.com/influxdata/influxdb/tsdb/store.go:231

1 @ 0x432fe0 0x442aeb 0xf7023a 0x460f61
# 0xf70239 github.com/influxdata/influxdb/services/precreator.(*Service).runPrecreation+0x109 /go/src/github.com/influxdata/influxdb/services/precreator/service.go:76

1 @ 0x432fe0 0x442aeb 0xfd25a0 0x460f61
# 0xfd259f github.com/influxdata/influxdb/vendor/go.opencensus.io/stats/view.(*worker).start+0xff /go/src/github.com/influxdata/influxdb/vendor/go.opencensus.io/stats/view/worker.go:154

1 @ 0x432fe0 0x443650 0x44363b 0x4433b7 0x484d9c 0x486467 0x4863f2 0xa1021d 0xa09e36 0xa37134 0xa15393 0xa29cf4 0xf7157f 0xf73301 0x460f61
# 0x4433b6 sync.runtime_SemacquireMutex+0x46 /usr/local/go/src/runtime/sema.go:71
# 0x484d9b sync.(*Mutex).lockSlow+0xfb /usr/local/go/src/sync/mutex.go:138
# 0x486466 sync.(*Mutex).Lock+0x96 /usr/local/go/src/sync/mutex.go:81
# 0x4863f1 sync.(*RWMutex).Lock+0x21 /usr/local/go/src/sync/rwmutex.go:98
# 0xa1021c github.com/influxdata/influxdb/tsdb.(*SeriesPartition).DeleteSeriesID+0x4c /go/src/github.com/influxdata/influxdb/tsdb/series_partition.go:318
# 0xa09e35 github.com/influxdata/influxdb/tsdb.(*SeriesFile).DeleteSeriesID+0x55 /go/src/github.com/influxdata/influxdb/tsdb/series_file.go:203
# 0xa37133 github.com/influxdata/influxdb/tsdb.(*Store).DeleteShard.func4+0x33 /go/src/github.com/influxdata/influxdb/tsdb/store.go:784
# 0xa15392 github.com/influxdata/influxdb/tsdb.(*SeriesIDSet).ForEach+0xb2 /go/src/github.com/influxdata/influxdb/tsdb/series_set.go:181
# 0xa29cf3 github.com/influxdata/influxdb/tsdb.(*Store).DeleteShard+0x4c3 /go/src/github.com/influxdata/influxdb/tsdb/store.go:783
# 0xf7157e github.com/influxdata/influxdb/services/retention.(*Service).run+0x1fe /go/src/github.com/influxdata/influxdb/services/retention/service.go:134
# 0xf73300 github.com/influxdata/influxdb/services/retention.(*Service).Open.func1+0x60 /go/src/github.com/influxdata/influxdb/services/retention/service.go:51

1 @ 0x4bde45 0x4bb8ba 0x4d4724 0x4d46e5 0x4dca11 0x4dc9e4 0x538971 0x537d58 0x4b086f 0x126b9c7 0x126b88a 0x126d821 0x1202183 0x1201a3f 0x1200acd 0x1200f8e 0x122978e 0x1227bed 0x1276317 0x460f61
# 0x4bde44 syscall.Syscall+0x4 /usr/local/go/src/syscall/asm_linux_amd64.s:18
# 0x4bb8b9 syscall.read+0x59 /usr/local/go/src/syscall/zsyscall_linux_amd64.go:732
# 0x4d4723 syscall.Read+0x163 /usr/local/go/src/syscall/syscall_unix.go:183
# 0x4d46e4 internal/poll.(*FD).Read+0x124 /usr/local/go/src/internal/poll/fd_unix.go:165
# 0x4dca10 os.(*File).read+0x70 /usr/local/go/src/os/file_unix.go:259
# 0x4dc9e3 os.(*File).Read+0x43 /usr/local/go/src/os/file.go:116
# 0x538970 bufio.(*Writer).ReadFrom+0xe0 /usr/local/go/src/bufio/bufio.go:725
# 0x537d57 bufio.(*Reader).WriteTo+0x277 /usr/local/go/src/bufio/bufio.go:498
# 0x4b086e io.copyBuffer+0x33e /usr/local/go/src/io/io.go:384
# 0x126b9c6 io.Copy+0x206 /usr/local/go/src/io/io.go:364
# 0x126b889 github.com/influxdata/influxdb/tsdb/engine/tsm1.(*directIndex).WriteTo+0xc9 /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/writer.go:430
# 0x126d820 github.com/influxdata/influxdb/tsdb/engine/tsm1.(*tsmWriter).WriteIndex+0xe0 /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/writer.go:722
# 0x1202182 github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Compactor).write+0x2e2 /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/compact.go:1174
# 0x1201a3e github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Compactor).writeNewFiles+0x18e /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/compact.go:1045
# 0x1200acc github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Compactor).compact+0x40c /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/compact.go:953
# 0x1200f8d github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Compactor).CompactFull+0x17d /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/compact.go:971
# 0x122978d github.com/influxdata/influxdb/tsdb/engine/tsm1.(*compactionStrategy).compactGroup+0x1b4d /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:2208
# 0x1227bec github.com/influxdata/influxdb/tsdb/engine/tsm1.(*compactionStrategy).Apply+0x4c /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:2185
# 0x1276316 github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).compactFull.func1+0x116 /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:2154

1 @ 0xa123a0 0xa33eef 0xa132e3 0xa11967 0xa11542 0xa33846 0x460f61
# 0xa1239f github.com/influxdata/influxdb/tsdb.(*SeriesPartitionCompactor).insertIDOffsetMap+0x2ff /go/src/github.com/influxdata/influxdb/tsdb/series_partition.go:696
# 0xa33eee github.com/influxdata/influxdb/tsdb.(*SeriesPartitionCompactor).compactIndexTo.func1+0x22e /go/src/github.com/influxdata/influxdb/tsdb/series_partition.go:616
# 0xa132e2 github.com/influxdata/influxdb/tsdb.(*SeriesSegment).ForEachEntry+0xf2 /go/src/github.com/influxdata/influxdb/tsdb/series_segment.go:250
# 0xa11966 github.com/influxdata/influxdb/tsdb.(*SeriesPartitionCompactor).compactIndexTo+0x326 /go/src/github.com/influxdata/influxdb/tsdb/series_partition.go:582
# 0xa11541 github.com/influxdata/influxdb/tsdb.(*SeriesPartitionCompactor).Compact+0x181 /go/src/github.com/influxdata/influxdb/tsdb/series_partition.go:538
# 0xa33845 github.com/influxdata/influxdb/tsdb.(*SeriesPartition).CreateSeriesListIfNotExists.func1+0xf5 /go/src/github.com/influxdata/influxdb/tsdb/series_partition.go:292

1 @ 0xab1075 0xab0e90 0xaadaba 0xaba5ca 0xabafe1 0xf56abc 0xf48acf 0x76cf54 0x7688f5 0x460f61
# 0xab1074 runtime/pprof.writeRuntimeProfile+0x94 /usr/local/go/src/runtime/pprof/pprof.go:708
# 0xab0e8f runtime/pprof.writeGoroutine+0x9f /usr/local/go/src/runtime/pprof/pprof.go:670
# 0xaadab9 runtime/pprof.(*Profile).WriteTo+0x3d9 /usr/local/go/src/runtime/pprof/pprof.go:329
# 0xaba5c9 net/http/pprof.handler.ServeHTTP+0x339 /usr/local/go/src/net/http/pprof/pprof.go:245
# 0xabafe0 net/http/pprof.Index+0x6f0 /usr/local/go/src/net/http/pprof/pprof.go:268
# 0xf56abb github.com/influxdata/influxdb/services/httpd.(*Handler).handleProfiles+0x8b /go/src/github.com/influxdata/influxdb/services/httpd/pprof.go:32
# 0xf48ace github.com/influxdata/influxdb/services/httpd.(*Handler).ServeHTTP+0x45e /go/src/github.com/influxdata/influxdb/services/httpd/handler.go:448
# 0x76cf53 net/http.serverHandler.ServeHTTP+0xa3 /usr/local/go/src/net/http/server.go:2802
# 0x7688f4 net/http.(*conn).serve+0x874 /usr/local/go/src/net/http/server.go:1890

1 @ 0xaf1730 0x460f61
# 0xaf1730 github.com/influxdata/influxdb/coordinator.(*PointsWriter).WritePointsPrivileged.func1+0x0 /go/src/github.com/influxdata/influxdb/coordinator/points_writer.go:311

```

Start by reproducing the long-running influxd workload and collecting a goroutine profile, then trace CreateSeriesListIfNotExists through tsdb/series_partition.go, tsdb/series_file.go, and tsdb/index/tsi1. Compare the relevant waits and locks with the write path in shard.go and coordinator/points_writer.go. Done means goroutines do not grow indefinitely and HTTP writes no longer degrade into repeated 500 timeouts.

Written by the indexing model from the issue text.