cockroachdb / cockroachdb/cockroach
import: subject import work to elastic CPU admission
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
(See the discussion in https://cockroachlabs.slack.com/archives/C01SRKWGHG8/p1701364715878499?thread_ts=1701013647.475009&cid=C01SRKWGHG8)
`importer.workloadReader.readFiles` and `bulk.BufferingAdder`, neither of which integrate with AC, were consuming 31% of the cpu on this node, which affected other user-facing queries.
AC has an existing pacing mechanism in `admission.Pacer` that can be used to throttle long running elastic CPU work (currently used for ExportRequest, TTL reads, rangefeed/changefeed initial and catchup scans). This seems like the right mechanism to use here. It can cause arbitrary delays, so it won't necessarily play well with timeouts at a higher layer -- I don't have enough understanding of the code surrounding `importer.ingestKvs` and `importer.workloadReader.readFiles` to know the pitfalls.

@yuzefovich @dt
Jira issue: CRDB-34026
Contributor guide
Research direction
Start by tracing importer.workloadReader.readFiles and bulk.BufferingAdder into importer.ingestKvs, then read the existing admission.Pacer uses for elastic CPU work. Determine how both import paths can use that pacing mechanism and how arbitrary delays interact with higher-layer timeouts. Done means import CPU work is subject to elastic admission without disrupting import timeouts.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100