[BUG] Orchestrator slow

Description
We observed an issue with the orchestrator, when it has to manage long states.
In details, the orchestrator runs a state that takes about 4 hours to complete (it performs a massive data extraction from Oracle and produces the full dataset into a Kafka topic); the orchestrator should wait the end of the state and then go on with the following orchestration step, if the state returns succesfully, or start the OnFail orchestration steps, if the state returns with an error status.
Actually it happens that, at some moment during the state execution, the minion stop temporary to return status messages to the master; so, the master reaches the timeout and considers the state returned with error, but actually the state is still running.
Consequently, the master takes the OnFail orchestration steps when there is not really need to be executed and this wrong behavior causes issues with the business process logic.
By looking at the salt-minion logs, it can be seen that the process hes produced no log entry for about six minutes, causing the timeout to be reached.
Note: the status that takes about 4 hours is executed by the orchestration in parallel with another status, that is much shorter.
Do you know if it is a bug or if we can take some action to avoid such a problem?

This is the log segment taken by Salt-minion that runs the long state, where you can see that between 17:16 and 17:22 the minion did non log anything. During this interval, it did not return any message to Salt master.

2021-05-05 17:16:25,352 [salt.minion      :2016][DEBUG   ][34517] minion return: {'success': True, 'return': {'pid': 114891, 'fun': 'state.sls', 'arg': ['rin-salt-orchestrator.orchestrator.kafka_streams.aligner_full_stock_integrator', {'__kwarg__': True, 'queue': False, 'concurrent': True}], 'tgt': 'prdrinkaf10', 'jid': '20210505121135424917', 'ret': '', 'tgt_type': 'glob', 'user': 'root'}, 'retcode': 0, 'jid': '20210505151607350458', 'fun': 'saltutil.find_job', 'fun_args': ['20210505121135424917']}
2021-05-05 17:16:25,646 [salt.utils.process:1055][DEBUG   ][98164] Subprocess ProcessPayload-Job-20210505151607350458 cleaned up
2021-05-05 17:22:59,432 [salt.minion      :1670][INFO    ][98164] User root Executing command saltutil.running with jid 20210505152241617890
2021-05-05 17:22:59,434 [salt.minion      :1674][DEBUG   ][98164] Command details {'fun': 'saltutil.running', 'arg': [], 'tgt': '*', 'jid': '20210505152241617890', 'ret': '', 'tgt_type': 'glob', 'user': 'root'}
2021-05-05 17:22:59,442 [salt.utils.process:1044][DEBUG   ][98164] Subprocess ProcessPayload-Job-20210505152241617890 added

Versions Report

Salt Version:
           Salt: 3002
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.8.1
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
         Jinja2: 2.8.1
        libgit2: Not Installed
       M2Crypto: 0.33.0
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.6.2
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 3.6.8 (default, Apr  2 2020, 13:34:55)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 17.0.0
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.1.4
 
System Versions:
           dist: centos 7 Core
         locale: UTF-8
        machine: x86_64
        release: 3.10.0-1062.9.1.el7.x86_64
         system: Linux
        version: CentOS Linux 7 Core

Start by tracing the orchestrator's long-running state.sls execution and the saltutil.find_job and saltutil.running status calls in the Salt master and minion logs. Reproduce a state lasting several hours with a six-minute status gap; done means the master waits for the actual result and does not run OnFail while the state is still active.