Spill transaction log data onto storage servers
- Dominant language
- C++
- Stars
- 16.7k
- Forks
- 1.6k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 126
Description
When enough data builds up on the transaction logs that we can no longer keep all of the unpopped mutations in memory, we need to write the mutations to disk. We call this process "spilling" the mutation from memory to disk.
Spilling puts an additional burden on the transaction logs that reduces their throughput significantly. The goal of this project is to have storage servers pull data from unpopped tags instead of letting the data stay on the transaction logs. Worst case, this will 2x increased write load on the storage servers. This plan also means we can spill for much longer without running out of disk space, and the disked used by the transaction logs can be sized to exactly fit memory.
There are two parts to implementing this. The first is a mechanism for telling the system to spill a tag to a set of storage servers:
* If no data is currently being spilled, set the startVersion to 0, otherwise use versionstampedkey as the startVersion
* The key also contains the tag that is being spilled
* The value is set to a vector of storage server IDs that will have the data
* ApplyMetadataMutation will take the message with the tags for those storage servers and private mutations that trigger the storage server to start copying data from that tag
* The spilled data is not stored in the main database keyspace so that it is not subject to data distribution moving it
* This means it is stored with a /xff/xff/xff prefix in the btree, and requires a new endpoint on the storage server to query the data
* All of the tlogs keep the map of where data is being spilled
* Peek requests in version ranges that are spilled provide the storage server ids
* The client caches a map of storage server ids to storage server interfaces, load balance is used to request version ranges
* pop requests are routed to a new endpoint on the storage servers
The second part is adding a component to data distribution which will spill tags for failed storage servers or regions.
Contributor guide
Research direction
Start by tracing ApplyMetadataMutation, peek and pop requests, the storage-server endpoint, and the data-distribution component described in the issue. Determine how spilled tags, storage-server IDs, version ranges, and the /xff/xff/xff keyspace should be represented and routed. Done means failed-server or region data can spill to storage servers and be read and popped without relying on transaction-log disk capacity.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100