apache / apache/couchdb

Set io_priority in all IO paths

Open
#4,101 4 comments 0 reactions 0 assignees View on GitHub
enhancement needs-triage
Dominant language
Erlang
Stars
7k
Forks
1.1k
Avg merge
1d 16h
Merged PRs (30d)
9

Description

## Summary

There are a number of codepaths performing IO operations that do not set an `io_priority` flag, so they do not properly get labeled or prioritized by IOQ and default to `other` io_class [1]. Many moons ago I made a PR [2] that went through and properly tagged anywhere I could find making IO requests without an `io_priority` set. It looks like the main gaps around indexing and indexer compaction were fixed in [3], however there are still a number of places lacking an `io_priority` value.

I also think it would be worthwhile to introduce at least one, perhaps two, additional IO classes. I think we should add a `system` io_priority class for things like auth cache, loading mem3 shards, global changes, couch_per_user, etc. I think it might also be worthwhile to add a dedicated `replication` io_priority, although it kind of looks like the replicator no longer does local db replications? In [2], the replicator still allowed for doing a direct `couch_changes` fetch of the local database, so perhaps my original concern around the replicator doing direct changes reads of full databases going unflagged with io_priority is no longer relevant. We should probably still tag the replicator doc updates with either `system` or `replication` though.

## Desired Behaviour

To minimize (or eliminate) the use of the fallback `other` io_priority class so that all IO is properly flagged with the appropriate IO class.

## Possible Solution

The PR in [2] has a number of simple additions of where to put the `io_priority` process dictionary value when missing. I tracked those down by modifying the code to throw an exception when absent and then iterated through all the locations I could find. I found many, but the approach was not fully exhaustive so I don't think we should crash by default when there's no `io_priority` set.

## Additional context

[1] https://github.com/apache/couchdb/blob/main/src/ioq/src/ioq.erl#L84-L85
[2] https://github.com/apache/couchdb/pull/1998/files
[3] https://github.com/apache/couchdb/commit/13bf0eb80813477bf3fbe79cffaf0faddffaaca9

Contributor guide

Open the contributing guide

Research direction

Start by reading the fallback behavior in src/ioq/src/ioq.erl and reviewing PR 1998 and commit 13bf0eb80813477bf3fbe79cffaf0faddffaaca9. Trace the remaining IO paths, including indexing, compaction, caches, and replication, to determine their appropriate classes. Done means IO operations no longer fall back to other and any added classes are consistently applied.

Written by the indexing model from the issue text.

Assessment

Tech stack
erlang
Domain
databases, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.