not an sstable (bad magic number)

We have found a scenario where we can create `not an sstable (bad magic number)` errors consistently on Linux while wrapping the library in the [JNA](https://github.com/java-native-access/jna) api. We believe we are using the LevelDB api correctly, though if you see an error in this test code, we'd love to know that's what's causing it.

We have experienced errors on every Linux we have tried. It seems to happen faster on virtualized Linux, but we've also run into errors while running Linux directly on metal. The magic sauce to getting a faster time to failure is setting the `-Xloggc` option on the JVM to a location that does not exist. We believe this is creating enough latency in the JVM to exacerbate a problem that is always present.

The following will complete (though, as we said, given enough time we believe it will still see corruption errors):

```
java -server -jar build/libs/leveldbtest-all.jar
```

The following will fail with corruption without a change in the code:

```
java -server -Xloggc:/does/not/exist -jar build/libs/leveldbtest-all.jar
```

With this issue report we are providing a gradle project and scripts for a test harness that is able to recreate this error in seconds by running on Docker (by far the worst time to failure we have found). While running in other configurations the time to failure is measured in hours (running on Azure or AWS) or days (`-Xloggc` pointing to a location that exists), but we are confident in saying that it's not a matter of if, but when the error will occur.

We would be more than happy to test any patches or answer your questions. We've spent weeks working on narrowing down this problem to its simplest example as it is critical to us that it is fixed.
## Versions used for the test
- Docker: 1.12.0
- Java: OpenJDK 1.8u102 (though we see this with Oracle's Java as well)
- JNA: 4.1.0
- LevelDB: 1.19
## Test error log

As you can see in the log excerpt below, the time to failure for this test harness was less than a second (sometimes it takes several seconds).

```
2016-09-14 15:30:38.162 [main ] INFO leveldb.test.LevelDbBlockStoreHarness: ---> Starting LevelDbBlockStoreHarness --->
2016-09-14 15:30:38.170 [main ] INFO leveldb.test.BlockDatabase: opening database: data/648960378880486400
2016-09-14 15:30:38.275 [main ] INFO leveldb.test.leveldb.LevelDBJNA: LevelDB version 1.19 loaded
2016-09-14 15:30:38.304 [LevelDbBlockStoreHarness-1 ] INFO leveldb.test.LevelDbBlockStoreHarness: Starting thread 1 of 1
2016-09-14 15:30:38.313 [LevelDbBlockStoreHarness-1 ] INFO leveldb.test.LevelDbBlockStoreHarness: Starting run 1 of 10 with database, BlockDatabase[dbDirectory = data/648960378880486400, totalBlockBytes = 0]
2016-09-14 15:30:39.034 [LevelDbBlockStoreHarness-1 ] ERROR leveldb.test.LevelDbBlockStoreHarness: Catching
leveldb.test.leveldb.api.CorruptionException: Corruption: not an sstable (bad magic number)
at leveldb.test.leveldb.api.Util.checkError(Util.java:20) ~[leveldbtest-all.jar:?]
at leveldb.test.leveldb.api.DB.write(DB.java:42) ~[leveldbtest-all.jar:?]
at leveldb.test.BlockDatabase.purgeIfFull(BlockDatabase.java:90) ~[leveldbtest-all.jar:?]
at leveldb.test.BlockDatabase.saveBlocks(BlockDatabase.java:68) ~[leveldbtest-all.jar:?]
at leveldb.test.LevelDbBlockStoreHarness$1.run(LevelDbBlockStoreHarness.java:74) [leveldbtest-all.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_102]
```
## Running the test
1. Install [Docker](https://www.docker.com/)
2. Unpack the provided zip file
3. Execute `leveldbtest/bin/run_docker.sh`
## What the test does

This test simply creates and removes random byte blobs from the database. The blobs are randomly sized, and written in random length batches. When the total bytes in the database exceeds 10M the blobs are deleted and the process continues.

The main method is in `leveldb.test.LevelDbBlockStoreHarness`. At the top of the file are a number of knobs you can turn to modify the execution of the test harness if you feel like during your testing.

``` java
// Number of databases to use concurrently
private static final int NUMBER_OF_DATABASES = 1;
// Number of threads to run concurrently
private static final int NUMBER_OF_THREADS = 1;
// Number of runs per thread
private static final int NUMBER_OF_RUNS = 10;
// Number of packages per run
private static final int NUMBER_OF_PACKAGES = 100;
// Minimum number of blocks per package (never set below 1)
private static final int MINIMUM_NUMBER_OF_BLOCKS = 1;
// Maximum number of blocks per package
private static final int MAXIMUM_NUMBER_OF_BLOCKS = 100;
// Minimum number of bytes per block (never set below 1)
private static final int MINIMUM_BLOCK_SIZE = 64 * 1024;
// Maximum number of bytes per block
private static final int MAXIMUM_BLOCK_SIZE = 256 * 1024;
```
## Notes
1. During our attempts to diagnose the error we recompiled LevelDB to run without using memory-mapped files. This had no effect on our ability to recreate this error.
2. We tested with multiple version of Ubuntu (12.04, 14.04 and 16.04) and with Oracle's Java and Open-JDK with no change in the ability to recreate this error beyond changing the mean time to failure.
## Test Harness

[sstable_corruption_test.zip](https://github.com/google/leveldb/files/473333/sstable_corruption_test.zip)