apple / apple/foundationdb

Understand the performance overhead on tLog spilling

Open
#2,431 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
16.7k
Forks
1.6k
Avg merge
1d 20h
Merged PRs (30d)
126

Description

tLog spilling's performance has been significantly improved, thanks to Alex's work in Issue: https://github.com/apple/foundationdb/issues/1048.

However, tLog spilling is not, and will not be, overhead free, compared to no-spilling scenario.
It is useful for system operators and users to understand how much performance degradation tLog spilling causes compared to no-spilling scenario.

This information can be useful for the following three scenarios:
1) When the remote cluster in the fearless setting is down (due to disaster or WAN network partition), tLog spilling will happen in the primary cluster. Understanding the performance degradation on the primary cluster can help the applications better prepare for the disaster scenario;

2) When applications use batch priority transactions to migrate data to a live cluster, batch priority transactions are throttled instead of spilled to disk when failures happen in the cluster. Allowing batch priority transaction to spill can help increase the migration speed, but it causes performance risks to normal transactions due to the spilling overhead. If the spilling overhead is not much, we may safely allow the spilling of batch priority transactions;

3) The new backup can cause tLog to spill to disk when the blob storage is not reachable, as discussed in Issue: https://github.com/apple/foundationdb/issues/2428#issuecomment-563472682.

The performance overhead of tLog spilling depends on many factors:
1) The ingesting workload load: how much writes and reads (in normal transactions and batch priority transactions) are coming into the cluster;
2) The number of tLogs that are spilling the mutations;
3) The hardware configurations, especially the CPU and disk on tLog.

Contributor guide

Open the contributing guide

Research direction

The issue names tLog spilling and references Issues 1048 and 2428, but no files, tests, or entry points. Start by reviewing those linked issues and the tLog spilling implementation, then define representative workload, tLog-count, and hardware comparisons whose completed measurements clearly quantify overhead against no spilling.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases, distributed-systems, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.