Additional tuning configs for scavenger jobs

Is your feature request related to a problem? Please describe.

My (self-hosted) cluster's persistence DB gets overwhelmed (100% cpu, high latency for ~20 mins at a time) on a cron schedule like clockwork, every 12 hours. I believe this is due to the scavenger jobs to clean up the DB. Unfortunately the only configs I have to tune this seems to be the various persistenceMaxQPS settings, which is too blunt of a tool.

The various scavengers also have inconsistent dynamic configuration available: the executions scanner has a per-shard qps setting, but the history scanner has no such configuration; its rps comes from persistenceMaxQPS

Describe the solution you'd like

I'd like to add a new dynamic configuration field for historyScannerRPS to tune this job. This will allow me to set a (low) limit for these heavy requests that won't overwhelm the underlying persistence DB, while keeping a higher limit for other persistence requests (so that the cluster's throughput isn't artificially limited just for these cleanup windows).

Another note: it may be useful to make the cron schedule configurable as well, defaulting to its current value. I don't need the ability to change this as the current cleanups happen mostly outside regular hours, but if i were to deploy to a different region where this cleanup schedule coincided with peak traffic, I'd have no options for mitigation.

Describe alternatives you've considered

The main alternative I've considered is lowering the persistenceMaxQPS setting. I've been careful about lowering this though as I don't want the cluster's throughput to suffer outside of the cleanup time. I've had to take this setting pretty low to see any difference in the load pattern on my DB.

Additional context

I've never made a contribution to this project, but would be willing to submit a PR if the project is inclined to accept one. I'm also open to alternative ideas; maybe there's something I'm overlooking and I could solve this in some other way.

Start with service/worker/scanner/workflow.go, including the existing executions-scanner per-shard setting and persistenceMaxQPS-derived RPS, then inspect service/worker/scanner/history/scavenger.go. Confirm how dynamic configuration reaches both scanner entry points. Done means history scavenger request rate can be tuned independently of the global persistence limit, with existing behavior preserved by default.