Minion scheduler not executed

Hey,

I have added a scheduler using the schedule.present state as follow:

midnight-execution:
  schedule.present:
    - persist: True
    - maxrunning: 1
    - function: state.apply
    - when: 00:02am

The state successfully run and I can see the schedule at the /etc/salt/minion.d/_schdule.conf file as follow:

schedule:
  __mine_interval: {enabled: true, function: mine.update, jid_include: true, maxrunning: 2, minutes: 60, return_job: false, run_on_start: true}
  midnight-execution: {enabled: true, function: state.apply, jid_include: true, maxrunning: 1, name: midnight-execution, when: '00:02am'}

Everything is fine until now...

The trouble come after few days when the scheduler stop working properly.

After several days/weeks the scheduler stop fully executing the midnight-execution, I can tell that using two methonds:

1. The new/updated functionality is not applied
2. The log

The NOT WORKING log file show the following:

[2021-08-18 23:09:00.016] [salt.utils.schedule] [INFO] Running scheduled job: __mine_inverval
[2021-08-19 00:02:00.015] [salt.utils.schedule] [INFO] Running scheduled job: midnight-execution
[2021-08-19 00:02:00.755] [salt.state] [INFO] Loading fresh modules for state activity
[2021-08-19 00:02:00.790] [salt.minion] [INFO] Returning information for job: req
[2021-08-19 00:02:00.791] [salt.minion] [INFO] Returning information for job: req
[2021-08-19 00:09:00.016] [salt.utils.schedule] [INFO] Running scheduled job: __mine_inverval
[2021-08-19 01:09:00.017] [salt.utils.schedule] [INFO] Running scheduled job: __mine_inverval
...
...
...
[2021-08-19 23:09:00.016] [salt.utils.schedule] [INFO] Running scheduled job: __mine_inverval
**The same routine all over again...**

This issue is not happening on all of my servers.
At WORKING servers I can see the following log:
This is how is suppose to be seen when working as expected

[2021-08-18 23:09:00.016] [salt.utils.schedule] [INFO] Running scheduled job: __mine_inverval
[2021-08-19 00:02:00.015] [salt.utils.schedule] [INFO] Running scheduled job: midnight-execution
[2021-08-19 00:02:00.755] [salt.state] [INFO] Loading fresh modules for state activity
[2021-08-19 00:02:00.799] [salt.template] [PROFILE] Time (in seconds) to render 'SOME TOP.SLS file path' using 'jinja' renderer: 0.0099799
[2021-08-19 00:02:00.799] [salt.template] [PROFILE] Time (in seconds) to render 'SOME TOP.SLS file path' using 'yaml' renderer: 0.0003178
...
[2021-08-19 00:02:03.799] [salt.fileclient] [INFO] Caching directory........
[2021-08-19 00:02:03.801] [salt.utils.extmods] [INFO] Syncing matchers for environment....
[2021-08-19 00:02:03.802] [salt.utils.extmods] [INFO] Loading cache from salt://_matchers, for environment ...
...
...
[2021-08-19 00:02:06.000] [salt.state] [INFO] Running state [STATE NAME] at time ....
[2021-08-19 00:02:06.005] [salt.state] [INFO] Executing state file.securse for [STATE NAME]
[2021-08-19 00:02:06.010] [salt.state] [INFO] The directory 'DIR PATH' is in the correct state
...

I'm running on Ubuntu 16.04:

• salt-master version 3000.6
• salt-minion version: 3000.2, 3000.3, 3002.2.

I have around ~1800 servers assign to the master and ~400 are having this issue (the scheduler is not executing the state.apply scheduled task).
The majority of the servers that are having this issue is running version 3002.2

Please help...
Best regards,
Amiram