Handling Missed Scheduled Reducer Executions

We currently do not persist the last execution time of scheduled reducers. As a result, events such as module updates, server restarts, or leader changes (anything that make SchedulerActor to restart) cause the next execution slot to be scheduled from "now" instead of the last known execution time.

This behavior is somewhat similar to tokio::time::MissedTickBehavior::Delay and seems reasonable when scheduled reducer actually miss their execution time slot due to any of the above mentioned events. However, it becomes unexpected in scenarios when SchedulerActor restarts qucikly and a scheduled reducer have not missed their execution time but drifted forward unneccesarily.

What should be the expected behavior?

• Keep it like tokio::time::MissedTickBehavior::Delay but do not drift reducer who has not missed their ticks.
• Make behaviour configurable by user?
• Let it be like thism, and update the same in documentation?

Start by reviewing the SchedulerActor scheduling flow and how scheduled reducer execution times are currently handled across restarts, updates, and leader changes. The expected missed-tick and drift behavior must be decided first; done would require an agreed behavior, implementation, and tests covering the restart scenarios described.