Set io_priority in all IO paths
- Dominant language
- Erlang
- Stars
- 7k
- Forks
- 1.1k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 9
Description
## Summary
There are a number of codepaths performing IO operations that do not set an `io_priority` flag, so they do not properly get labeled or prioritized by IOQ and default to `other` io_class [1]. Many moons ago I made a PR [2] that went through and properly tagged anywhere I could find making IO requests without an `io_priority` set. It looks like the main gaps around indexing and indexer compaction were fixed in [3], however there are still a number of places lacking an `io_priority` value.
I also think it would be worthwhile to introduce at least one, perhaps two, additional IO classes. I think we should add a `system` io_priority class for things like auth cache, loading mem3 shards, global changes, couch_per_user, etc. I think it might also be worthwhile to add a dedicated `replication` io_priority, although it kind of looks like the replicator no longer does local db replications? In [2], the replicator still allowed for doing a direct `couch_changes` fetch of the local database, so perhaps my original concern around the replicator doing direct changes reads of full databases going unflagged with io_priority is no longer relevant. We should probably still tag the replicator doc updates with either `system` or `replication` though.
## Desired Behaviour
To minimize (or eliminate) the use of the fallback `other` io_priority class so that all IO is properly flagged with the appropriate IO class.
## Possible Solution
The PR in [2] has a number of simple additions of where to put the `io_priority` process dictionary value when missing. I tracked those down by modifying the code to throw an exception when absent and then iterated through all the locations I could find. I found many, but the approach was not fully exhaustive so I don't think we should crash by default when there's no `io_priority` set.
## Additional context
[1] https://github.com/apache/couchdb/blob/main/src/ioq/src/ioq.erl#L84-L85
[2] https://github.com/apache/couchdb/pull/1998/files
[3] https://github.com/apache/couchdb/commit/13bf0eb80813477bf3fbe79cffaf0faddffaaca9
Contributor guide
Research direction
Start by reading the fallback behavior in src/ioq/src/ioq.erl and reviewing PR 1998 and commit 13bf0eb80813477bf3fbe79cffaf0faddffaaca9. Trace the remaining IO paths, including indexing, compaction, caches, and replication, to determine their appropriate classes. Done means IO operations no longer fall back to other and any added classes are consistently applied.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- erlang
- Domain
- databases, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100