apple / apple/foundationdb

Client-side priority improvements

Open
#4,327 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
16.7k
Forks
1.6k
Avg merge
1d 20h
Merged PRs (30d)
126

Description

The prioritization of requests on clients needs to be carefully considered. A quick summary of the current state of things:

1. Incoming requests on typical clients are prioritized at `DefaultOnMainThread=7500`.
2. Most request responses are prioritized at `DefaultPromiseEndpoint=8000`.
3. Database objects have an associated priority, which for normal clients is `DefaultEndpoint=5000`. This is used in various delays, etc., on the client, but it doesn't affect the prioritization of incoming requests nor the prioritization of handling responses from the cluster.
4. Transactions also have an associated priority which they typically inherit from the database, but sometimes this is set explicitly (e.g. for data distribution).
5. One exception to the above is the GRV request, for which replies from the cluster are handled at the database priority (default 5000).
6. There are some places throughout the client that don't follow the above rules and use other priorities.

The prioritization of replies from the cluster over new requests seems sensible, as does the choice to prioritize requests for transactions that have already started instead of for starting new ones. However, other aspects of the prioritization are a bit confusing, and it's not clear if the current behavior was carefully intended or happened somewhat by accident.

As usual, toying with priorities is a tricky game, so changes here need to be done thoughtfully and with good testing. Some ideas for things to look into:

1. GRV requests are prioritized below others, but only at the point when the client receives a reply. At this point we are already on the 5s transaction clock. Should we instead decrease priority before sending a batch and be more timely with responses?
2. The delineation between database priority and the priorities of incoming requests and responses is a little confusing, and possibly this behavior was not intentional. We should clarify what this interaction should be.
3. We should evaluate any place where we aren't following the above priority rules (e.g. any delay that uses default or some other arbitrarily chosen priority, etc.).
4. Is there any benefit to relative prioritization of request types in addition to GRV? For example, prioritizing reads over commits (reads are expected to be faster than commits and are subject to the 5s clock, an answered commit is no longer on the clock). The starvation behavior may be worse here, and possibly that's reason to keep things as is.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the client paths for incoming requests, cluster replies, GRV handling, database priorities, transaction priorities, and delays that use other values. Compare their behavior with the proposed priority rules, then define and test a consistent policy for GRV, reads, commits, and transaction-starting requests.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.