microsoft / microsoft/FASTER

Can we have synchronous API for log.ReadAsync

Open
#932 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C#
Stars
6.6k
Forks
595
PR merge metrics
No merged PRs in 30d

Description

Thank you so much for this amazing project!

I’m currently working on building an in-memory database that can handle datasets larger than the available RAM.
For persistence, I’m using Faster Log, which seems very promising. The concurrency control relies on the ReaderWriterLockSlim, which keeps the implementation simple, particularly for synchronous APIs (without using async/await).

The challenge appears when considering how to fetch data from the log once the data record has been flushed to disk (to maintain a small memory footprint). I noticed that the log only provides an asynchronous method for this:

(result, length) = await log.ReadAsync(iter.CurrentAddress);

However, this creates an issue for my implementation, which relies on ReaderWriterLockSlim. It breaks serializable execution because I can’t use async/await within a transaction.

Having a synchronous version of log.ReadAsync would be incredibly helpful. That way, during a transaction, I could verify that a record is missing from RAM but still fetch it from disk using its address.

Maybe I can use log.ReadAsync(iter.CurrentAddress).GetAwaiter().GetResult(), but I am not sure about deadlocks that may happen with such an approach.

I would appreciate any suggestions.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the log.ReadAsync entry point and trace how reads behave when records have been flushed to disk. Review the interaction with ReaderWriterLockSlim and determine the synchronous API's deadlock behavior; done means callers can retrieve a record by address during a synchronous transaction without async/await.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
database
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.