Segmentation fault in db_bench when num_multi_db > 1

We are running this benchmark on a dual socket 24 core AMD EPYC server. Attempts to reproduce on a lower core count machine (VM with 2 CPUs) did not result in a segfault.

### Expected behavior
slow db_bench run in debug mode

### Actual behavior
db_bench segfaults on cleanup in sharded_cache.cc

### Steps to reproduce the behavior
Checkout tag "v5.18.3"
make db_bench
run:
./db_bench --compression_type=none --compression_ratio=0.75 --benchmarks=fillseq --histogram true --num 100000 -block_size 1048576 --compression_level=6 --db=/benchmark/rocksdb_bench/db1 -key_size 20 -value_size 4096 -num_low_pri_threads 8 -num_multi_db 4 -threads 12

This does not always result in a segmentation fault but on our system fails reliably within a few runs.

Full output of the db_bench run with addr2line run on stack trace:
./db_bench --compression_type=none --compression_ratio=0.75 --benchmarks=fillseq --histogram true --num 100000 -block_size 1048576 --compression_level=6 --db=/benchmark/rocksdb_bench/db1 -key_size 20 -value_size 4096 -num_low_pri_threads 8 -num_multi_db 4 -threads 12
Initializing RocksDB Options from the specified file
Initializing RocksDB Options from command-line flags
RocksDB: version 5.18
Date: Thu Feb 14 11:09:18 2019
CPU: 96 * AMD EPYC 7401 24-Core Processor
CPUCache: 512 KB
Keys: 20 bytes each
Values: 4096 bytes each (3072 bytes after compression)
Entries: 100000
Prefix: 0 bytes
Keys per prefix: 0
RawSize: 392.5 MB (estimated)
FileSize: 294.9 MB (estimated)
Write rate: 0 bytes/second
Read rate: 0 ops/second
Compression: NoCompression
Memtablerep: skip_list
Perf Level: 1
WARNING: Assertions are enabled; benchmarks unnecessarily slow
------------------------------------------------
Initializing RocksDB Options from the specified file
Initializing RocksDB Options from command-line flags
DB path: [/benchmark/rocksdb_bench/db1]
fillseq : 3.444 micros/op 290394 ops/sec; 1139.9 MB/s
Microseconds per write:
Count: 4800000 Average: 41.0202 StdDev: 493.69
Min: 4 Median: 32.6538 Max: 127232
Percentiles: P50: 32.65 P75: 45.81 P99: 134.13 P99.9: 306.27 P99.99: 15286.71
------------------------------------------------------
( 3, 4 ] 11 0.000% 0.000%
( 4, 6 ] 15923 0.332% 0.332%
( 6, 10 ] 118135 2.461% 2.793%
( 10, 15 ] 289731 6.036% 8.829% #
( 15, 22 ] 893653 18.618% 27.447% ####
( 22, 34 ] 1219340 25.403% 52.850% #####
( 34, 51 ] 1531033 31.897% 84.746% ######
( 51, 76 ] 592719 12.348% 97.095% ##
( 76, 110 ] 74279 1.547% 98.642%
( 110, 170 ] 42702 0.890% 99.532%
( 170, 250 ] 15783 0.329% 99.861%
( 250, 380 ] 4369 0.091% 99.952%
( 380, 580 ] 1297 0.027% 99.979%
( 580, 870 ] 124 0.003% 99.981%
( 870, 1300 ] 51 0.001% 99.982%
( 1300, 1900 ] 60 0.001% 99.984%
( 1900, 2900 ] 50 0.001% 99.985%
( 2900, 4400 ] 28 0.001% 99.985%
( 4400, 6600 ] 28 0.001% 99.986%
( 6600, 9900 ] 147 0.003% 99.989%
( 9900, 14000 ] 34 0.001% 99.990%
( 14000, 22000 ] 143 0.003% 99.992%
( 22000, 33000 ] 108 0.002% 99.995%
( 33000, 50000 ] 84 0.002% 99.996%
( 50000, 75000 ] 132 0.003% 99.999%
( 75000, 110000 ] 24 0.001% 100.000%
( 110000, 170000 ] 12 0.000% 100.000%

Received signal 11 (Segmentation fault)
#0 ./db_bench(+0xa055c) [0x55580a30655c] ?? ??:0
#1 ./db_bench(+0x2c535f) [0x55580a52b35f] ?? ??:0
#2 ./db_bench(+0x2c53d3) [0x55580a52b3d3] ?? ??:0
#3 ./db_bench(+0x2c5ab1) [0x55580a52bab1] ?? ??:0
#4 ./db_bench(+0x9e1f4) [0x55580a3041f4] ?? ??:0
#5 ./db_bench(+0x1e67c8) [0x55580a44c7c8] ?? ??:0
#6 ./db_bench(+0x146193) [0x55580a3ac193] ?? ??:0
#7 ./db_bench(+0x1311d3) [0x55580a3971d3] ?? ??:0
#8 ./db_bench(+0x1316fc) [0x55580a3976fc] ?? ??:0
#9 ./db_bench(+0x354574) [0x55580a5ba574] ?? ??:0
#10 ./db_bench(+0x354707) [0x55580a5ba707] ?? ??:0
#11 /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xbd57f) [0x7fddcfb5357f] ?? ??:0
#12 /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7fddd0d376db] ?? ??:0
#13 /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7fddcf21088f] ?? ??:0
Segmentation fault (core dumped)

addr2line -afp -e db_bench a055c
0x00000000000a055c: _ZN7rocksdb12ShardedCache5EraseERKNS_5SliceE at /home/users/dsloan/repos/RocksDB-Orig/cache/sharded_cache.cc:75

addr2line -afp -e db_bench 2c535f
0x00000000002c535f: _ZN7rocksdb15BlockBasedTable5CloseEv at /home/users/dsloan/repos/RocksDB-Orig/table/block_based_table_reader.cc:3052

addr2line -afp -e db_bench 2c53d3
0x00000000002c53d3: _ZN7rocksdb15BlockBasedTableD2Ev at /home/users/dsloan/repos/RocksDB-Orig/table/block_based_table_reader.cc:65

addr2line -afp -e db_bench 2c5ab1
0x00000000002c5ab1: _ZN7rocksdb15BlockBasedTableD0Ev at /home/users/dsloan/repos/RocksDB-Orig/table/block_based_table_reader.cc:66

addr2line -afp -e db_bench 9e1f4
0x000000000009e1f4: _ZN7rocksdb13LRUCacheShard5EraseERKNS_5SliceEj at /home/users/dsloan/repos/RocksDB-Orig/./cache/lru_cache.h:113

addr2line -afp -e db_bench 1e67c8
0x00000000001e67c8: _ZN7rocksdb10TableCache5EvictEPNS_5CacheEm at /home/users/dsloan/repos/RocksDB-Orig/db/table_cache.cc:474

addr2line -afp -e db_bench 146193
0x0000000000146193: _ZN7rocksdb6DBImpl18PurgeObsoleteFilesERNS_10JobContextEb at /home/users/dsloan/repos/RocksDB-Orig/db/db_impl_files.cc:463

addr2line -afp -e db_bench 1311d3
0x00000000001311d3: _ZN7rocksdb6DBImpl19BackgroundCallFlushEv at /home/users/dsloan/repos/RocksDB-Orig/db/db_impl_compaction_flush.cc:2100

addr2line -afp -e db_bench 1316fc
0x00000000001316fc: _ZN7rocksdb6DBImpl11BGWorkFlushEPv at /home/users/dsloan/repos/RocksDB-Orig/db/db_impl_compaction_flush.cc:1918

addr2line -afp -e db_bench 354574
0x0000000000354574: _ZN7rocksdb14ThreadPoolImpl4Impl8BGThreadEm at /usr/include/c++/7/bits/std_function.h:275

addr2line -afp -e db_bench 354707
0x0000000000354707: _ZN7rocksdb14ThreadPoolImpl4Impl15BGThreadWrapperEPv at /home/users/dsloan/repos/RocksDB-Orig/util/threadpool_imp.cc:305

Build and run db_bench using the supplied v5.18.3 checkout and command, then inspect cleanup around cache/sharded_cache.cc:75. Trace the call path through table/block_based_table_reader.cc:3052, db/table_cache.cc:474, and the background flush and obsolete-file purge locations; done means the reproduction no longer segfaults during cleanup.

Written by the indexing model from the issue text.