linksplatform / linksplatform/Data

MapReduce multithread combined storage

Open
#77 2 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
C#
Stars
1
Forks
3
PR merge metrics
No merged PRs in 30d

Description

We have an ability to set minimum value of internal references range.

https://github.com/linksplatform/Data/blob/25bcf2c1e26c90e9573df895ebf234d2913c8a0d/csharp/Platform.Data/LinksConstants.cs#L138

We also can allow to reference any address even if it does not exist in the current storage.

Then we can create separate file or allocate separate section for each new N links.

Later, we will be able to combine all read and write operations on the combined storage.

Each section of links storage can be allocated in separate heap block or in separated file. And can be always accessed by specific thread. That means that all read and write operations can be spread across multiple threads, and we can distribute the load this way without using any locks at all, just lock free queues to stream requests and results to and from threads.

Each request is mapped to all threads, when results are ready they are reduced to a single result.

In the case of heap allocation, each section (64 MB or any user-defined size) can be allocated separately without need to copy data, so this can save additional CPU resources.

In the case of `mmap` allocation, there is no need to close files (just new ones will be open), so there is no need to force flush data to disk, this it also saves resources during regular operation of the storage.

All trees in all sections are smaller, so it also helps to scale.

No need to use more than C+1 threads, where C is the number of memory channels in the system.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing csharp/Platform.Data/LinksConstants.cs around line 138 and the existing storage implementation. Define the concrete scope and acceptance criteria for sectioned storage, multithreaded reads and writes, and MapReduce-style result reduction; the issue does not name tests or a specific entry point.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.