JuliaData / JuliaData/MemPool.jl

Add transactions

Open
#71 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement lru performance storage
Dominant language
Julia
Stars
25
Forks
16
Avg merge
4h 56m
Merged PRs (30d)
1

Description

At the moment, our storage layer (in StorageState) is non-transactional, and so it's not possible to ensure that a certain set of operations occur without other (incompatible or performance-degrading) operations occurring in-between. This can make it impossible for the SimpleRecencyAllocator (which provides our swap-to-disk functionality) to be able to provide any guarantees that memory/disk usage actually falls within user-provided limits.

The obvious solution here is to add "transactions", where the set of operations to be performed in sequence are batched up and submitted as a single logical operation. Any transaction in progress can exclude other operations or transactions from occurring on the same piece(s) of data until the current transaction completes. Additionally, for performance, we can implement some logic to "merge" compatible operations across multiple transactions (or within the same transaction) when the end result will be identical to running the operations exclusively.

@dbedi3311

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the StorageState storage layer and the SimpleRecencyAllocator swap-to-disk component. Clarify the transaction API, exclusion semantics, and compatible-operation merging with maintainers before implementation; done should include guarantees that memory and disk usage remain within user-provided limits.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.