RocksIterator can not get the data on secondary instance

> Note: Please use Issues only for bug reports. For questions, discussions, feature requests, etc. post to dev group: https://groups.google.com/forum/#!forum/rocksdb or https://www.facebook.com/groups/rocksdb.dev

One of Rocksdb secondary instance can not get the correct data after 16 hours.

### Expected behavior
startup one primary and 3 secondary instances on k8s. every instances can get the same result for the same query.

### Actual behavior

after 16 hours, primary and 2 secondary instances got 1000 records for user. one instance got 500 records.

### Steps to reproduce the behavior
1. startup one primary and 3 secondary instances on k8s.
2. secondary instances invoke tryCatchUpWithPrimary every 3 seconds.
3. everything wroks fine at this time, can get the items correctly on the 3 secondary instances
4. after one day, there is a secondary instance become abnormal, can not get the whole data comparing to other instances, no exception throwed by tryCatchUpWithPrimary. example: other 3 instance(primary and other 2 secondary) can get 1000 records, but this instance can only get 500 records. and can not get the newest item inserted by primary as well

first saw on Java RocksdbJni 8.8.1, then downgrade to 6.29.4, still the same.
env:
Java + Spring Boot
aws k8s/aws efs
Rocksdb version: 6.29.4
Java RocksdbJni: 6.29.4.1

have any suggestion for this issue?

No file or test is named. Start with the RocksIterator reads and the tryCatchUpWithPrimary entry point, then reproduce the one-primary/three-secondary Kubernetes setup over the reported time period and compare the instances when one returns 500 records. Done means explaining and correcting the secondary's divergence so all instances see the same and newest data.

Written by the indexing model from the issue text.