hiero-ledger / hiero-ledger/hiero-consensus-node

Database should use Async file APIs

Open
#5,201 11 comments 0 reactions 1 assignee Assigned to @jasperpotts View on GitHub
Epic Improvement P2 Performance Platform Platform Data Structures
Dominant language
Java
Stars
406
Forks
226
Avg merge
3d 4h
Merged PRs (30d)
210

Description

All the Java APIs for disk access use the older synchronous APIs for disk IO. We know from testing there is a 2-10x gain in IOPS if we move to a async API like AIO or IO_URING. Nether of these are available in Java out of the box so we will need to develop a custom native solution. Maybe an alternative implementation of [AsynchronousFileChannel](https://docs.oracle.com/javase/7/docs/api/java/nio/channels/AsynchronousFileChannel.html) might work as a API (Via a new FileSystem Implementation though java.nio.file.spi might work).

We don't know when we are going to need that performance boost but this is a good candidate to get a large boost.

We have tested all the available File APIs in the JDK and not found anything faster than what we are doing with FileChannels. The next step will be to look at using a better low level Async OS API. Such as IO_Uring or AIO, connected to Java with JNI or Project Panama.

Oleg had a good suggestion for a phased plan:

### Phase 1 - Add Async Base API
Design and create a new async API that can be used by lowest level of database. It can be implemented either using java `AsynchronousFileChannel` or just a thread pool over file channels. But should be designed so that it can be implemented with LibAIO or IO_Uring native API. Update base part of the DB/VM like compactions and hashing to use new APIs. It might make sense for this API to be in PBJ IO so that it can be fully integrated and work natively in PBJ types like `Bytes`.

### Phase 2 - Prototype implementation using IO_Uring and/or LibAIO
Pick one or more native AIO library and implement prototype implementations, maybe try multiple and benchmark on a linux server with main net spec.

### Phase 3 - Extend Async API usage up to App Layer
Expose Async API up to VirtualMap layer so it can be used by the App if we think there is performance benefit.

### Phase 4 - Final implementation of native code for IO_Uring etc.
Make the chosen native implementation production ready.

### Formally bug was:

> DataFileReader is used from a large number of threads concurrently:
>
> - 20+ Hashing Threads
> - Pre-handle get for modify threads
> - Merging Threads
> - Reconnect Threads
> - Pipeline thread
>
> etc. It used to handle this well before we fixed an issue found by Sonar with ThreadLocal handling. We used to create a new `FileChannel` for each thread and file pair. This was great for performance but would leek file handles over time as files came and went. With thread-local file channels there is no nice way to find every thread that has ever read from a given file when they file is deleted so that we can clean up and close those file channels.
>
> So the way we do it today is to share one `FileChannel` per file between all threads that read from that file. On the surface that looked like a great solution as `FileChannel` and the underlying `pread()` unix command it uses support multiple concurrent reads. The problem is the `FileChannel` uses a synchronized method, this means all the reading threads fight over that lock, causing huge contention and slowing things down by at least half.
>
> So the challenge is how to support multiple unknown threads to read from a file concurrently. The nice solution seems to be to memory-map the files which will allow multiple readers and offer other performance benefits. The problem is the Java memory-map API has limitations, the biggest of which is there is no API for unmapping a file. There is a workaround using sun.misc.Unsafe, which might work and is often used it just needs **VERY** careful handling or it can cause seg-faults. Even if we work around that problem then we hit the problem that ByteBuffer can only address 2GB of a file at a time because it's indexes are integers. This can be worked around by limiting the max file size to 2GB or by using multiple ByteBuffer mapping to map 2Gb regions of the file.
>
> The nicer solution seems to be to use the new Java 17 - JEP 393: Foreign-Memory Access API (Third Incubator). It supports mapping files larger than 2Gb, controlled unmapping and safe multithreaded access. This will need to wait till we are building Swirlds with Java 17.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.