Durable Jobs follow-up
- Dominant language
- C#
- Stars
- 10.9k
- Forks
- 2.1k
- Avg merge
- 15h 1m
- Merged PRs (30d)
- 345
Description
This issue documents follow-up items for Durable Jobs (#9717)
* [x] **Rename** to Durable Jobs (update projects, types, etc)
* [X] Rename `ScheduledJobContext` to `ScheduledJobRun` #9798
* [ ] Address 50K append/blob limit for Azure Storage implementation (https://github.com/dotnet/orleans/pull/9717#pullrequestreview-3394719873)
* [x] Use `IOverloadDetector` in `LocalScheduledJobManager` to throttle execution when the host is overloaded (https://github.com/dotnet/orleans/pull/9717#pullrequestreview-3394850102)
* [X] `IScheduledJobReceiverExtension.DeliverScheduledJobAsync` should return some kind of `ScheduledJobRunResult` which includes a `TimeSpan` `PollAfter` property so that long-running requests can be better supported. #9802
* [X] Flow `CancellationToken` in tests, so we can have a short (eg, 2-min) timeout for each test. #9909
* [X] Make the non-`CancellationToken` arguments passed to `ILocalScheduledJobManager.ScheduleJobAsync` a class or struct to make it easier to add properties later without breaking existing callers/impls. #9908
* [ ] Idea: Support **multiple concurrent accounts** in `AzureStorageJobShardManager` for improved scaling, migration, etc (https://github.com/dotnet/orleans/pull/9717#pullrequestreview-3404983204)
* [ ] **Observability** - Do a pass on tracing, metrics, logs
* [ ] **Rebalancing** - We need to de-assign/rebalance shards if we have too many to avoid skew after a new deployment or upgrade. (https://github.com/dotnet/orleans/pull/9717#pullrequestreview-3405930327)
* [ ] **Concurrent shard limit** - We should limit the number of concurrently assigned shards per silo to prevent memory exhaustion. (https://github.com/dotnet/orleans/pull/9717#pullrequestreview-3405930327)
* [ ] **Shard assignment slow start** - We should consider performing a slow start for shard assignment, only reading a number of shards based on how long the silo has been up. It's important for disaster recovery scenarios, as we have seen with Azure ML. #9943
* [X] **Concurrent job slow start** - We should gradually increase job concurrency (semaphore.Release) during startup until we hit our target. This helps to avoid starvation issues which can happen before things have warmed up (caches, connection pools, thread pool sizing, etc). #9910
* [ ] Make sure we handle **multi-cluster deployments** more gracefully in AzureStorageJobShardManager (and other impls, ideally) (https://github.com/dotnet/orleans/pull/9717#pullrequestreview-3404974478)
Contributor guide
Assessment
This issue has not been assessed yet.