Improve Feedback After Failed Panorama Commit Jobs

Is your feature request related to a problem?

If Panorama pushes new configuration to a managed firewall, and the Panorama commit succeeds but the subsequent firewall commit fails, the result is an error message which describes a timeout, not a failed commit. For example, when a firewall commit initiated by Panorama failed, this error was received:

.....panos/base.py\", line 4705, in syncjob\n    \"Timeout waiting for \" + \"job %s completion\" % job\npan.xapi.PanXapiError: Timeout waiting for job 536 completion\n"

Describe the solution you'd like

I'm no Python expert, but I assume that the current implementation waits for an arbitrary time, and if the Panorama job is not observed as completed by the timer expiring, it is assumed timed out. Then we get the error currently on line 4721 in base.py:

raise pan.xapi.PanXapiError(
    "Timeout waiting for " + "job %s completion" % job
)

Ideally, the preceding code would check if the problem is a real timeout in communication or connectivity between the relevant components, or if the problem is a downstream commit fail on the firewall itself, after Panorama attempts to push new configuration to that firewall.

Describe alternatives you've considered

One could alternatively: 1) attempt the changes via Panorama, then 2) after receiving success or timed out from the Python code, try to check the firewall and manually check if the firewall commit was successful. However, ensuring you're checking the same commit job that Panorama instigated would be potentially difficult. Also, this seems like something which would ideally be fixed within the commit procedure in this SDK, not something one has to put in their own code, especially given this particular error instance was observed whist using the Ansible modules that wrap around this SDK.

Additional context

This error instance, where what was reported as a commit timeout was actually a commit fail, was observed whilst executing an Ansible playbook. Here is the full error detail:

TASK [Commit] ***********************************************************************************************************************************************************************************

An exception occurred during task execution. To see the full traceback, use -vvv. The error was: pan.xapi.PanXapiError: Timeout waiting for job 536 completion

fatal: [panorama]: FAILED! => {"changed": false, "module_stderr": "Traceback (most recent call last):\n  File \"/home/vagrant/.ansible/tmp/ansible-tmp-1617784945.4697285-3010-161434147552941/AnsiballZ_panos_commit_push.py\", line 102, in <module>\n    _ansiballz_main()\n  File \"/home/vagrant/.ansible/tmp/ansible-tmp-1617784945.4697285-3010-161434147552941/AnsiballZ_panos_commit_push.py\", line 94, in _ansiballz_main\n    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n  File \"/home/vagrant/.ansible/tmp/ansible-tmp-1617784945.4697285-3010-161434147552941/AnsiballZ_panos_commit_push.py\", line 40, in invoke_module\n    runpy.run_module(mod_name='ansible_collections.paloaltonetworks.panos.plugins.modules.panos_commit_push', init_globals=None, run_name='__main__', alter_sys=True)\n  File \"/usr/lib64/python3.6/runpy.py\", line 205, in run_module\n    return _run_module_code(code, init_globals, run_name, mod_spec)\n  File \"/usr/lib64/python3.6/runpy.py\", line 96, in _run_module_code\n    mod_name, mod_spec, pkg_name, script_name)\n  File \"/usr/lib64/python3.6/runpy.py\", line 85, in _run_code\n    exec(code, run_globals)\n  File \"/tmp/ansible_panos_commit_push_payload_vh_qtedl/ansible_panos_commit_push_payload.zip/ansible_collections/paloaltonetworks/panos/plugins/modules/panos_commit_push.py\", line 201, in <module>\n  File \"/tmp/ansible_panos_commit_push_payload_vh_qtedl/ansible_panos_commit_push_payload.zip/ansible_collections/paloaltonetworks/panos/plugins/modules/panos_commit_push.py\", line 183, in main\n  File \"/home/vagrant/ansible/lib/python3.6/site-packages/panos/base.py\", line 4482, in commit\n    sync=sync, exception=exception, cmd=cmd, admins=admins, sync_all=sync_all\n  File \"/home/vagrant/ansible/lib/python3.6/site-packages/panos/base.py\", line 4583, in _commit\n    result = self.syncjob(commit_response, sync_all=sync_all)\n  File \"/home/vagrant/ansible/lib/python3.6/site-packages/panos/base.py\", line 4705, in syncjob\n    \"Timeout waiting for \" + \"job %s completion\" % job\npan.xapi.PanXapiError: Timeout waiting for job 536 completion\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}

PLAY RECAP **************************************************************************************************************************************************************************************
panorama                   : ok=2    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

Job 536 had not timed out, there was no "time out" of communication or connectivity between the host running Ansible and Panorama, or between Panorama and the managed firewall. The job had generated config which was valid at Panorama, but once attempted to be committed on the firewall caused a validation fail. Panorama showed this output for the Ansible execution shown above:

admin@Panorama> show jobs id 536

Enqueued              Dequeued           ID                              Type                         Status Result Completed
------------------------------------------------------------------------------------------------------------------------------
2021/04/07 01:45:36   01:45:36          536                         CommitAll                            FIN     OK 100 %

Description:
 -  441113F4BC075AB          commit timed out   FAIL 01:45:36

It seems Panorama describes a failed commit on the firewall as "timed out" too, unfortunately.

Start in panos/base.py at syncjob and the timeout error around line 4721. Inspect how Panorama reports downstream firewall commit failures versus communication timeouts, then verify that the reported job status produces a failure-specific error rather than a timeout; the issue does not name a test file.