"Too many open files" even though the system limits are not reached

We are seeing this error sometimes (once in a few months). We get the following exception:
```
Caused by: org.rocksdb.RocksDBException: While open a file for random read: /rocks/database/1671607.sst: Too many open files
at org.rocksdb.RocksDB.put(Native Method)
at org.rocksdb.RocksDB.put(RocksDB.java:716)
```

All our investigations show that the number of file descriptors we have open are much less than the limits. Upon catching this exception we are calling `lsof` and saving the output, which shows that there were ~160K file handles. The system limit is set to ~655K.
```
$ ulimit -Hn
655350
$ ulimit -Sn
655350
$ cat /proc/9221/limits | grep "open files"
Max open files 655350 655350 files
```

We also ran some test programs (with lower limits set) and we have verified that limits are being honored. I don't see how rocksdb would be reaching anywhere close to the 655K limit though.

Our database currently has 570 sst files. `max_open_files` is set to -1 and we are using universal compaction. My understanding is that this means the number of sst files can double during compaction, which brings us to 1140 files.

We are using rocksdb version 6.13.3.