SAS Viya 3.5 deployment failed - known issue with Ansible, still an issue.

https://azure.microsoft.com/en-ca/resources/templates/sas-viya/

### Issue Details

### Repro steps (*if necessary, delete otherwise*).
1. downloaded the SAS Viya 3.5 package and uploaded into Azure blob.
2. Followed the instructions : https://azure.microsoft.com/en-ca/resources/templates/sas-viya/
3. Failed the process after 2 hours.
4. Contacted Microsoft and SAS.
**5. SAS clarified that it's a know issue, still it's an issue after the recent push to repository. Please help on this.**

log details:
{"code":"DeploymentFailed","message":"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.","details":[{"code":"Conflict","message":"{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource operation completed with terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"VMExtensionProvisioningError\",\r\n \"message\": \"VM has reported a failure when processing extension 'AnsibleControllerStartup'. Error message: \\\"Enable failed: failed to execute command: command terminated with exit status=2\\n[stdout]\\nendpoint URL=https://services.viya.sas:443/\\n\\nTASK [dagentsrv : Save sas endpoint URL] ***************************************\\nok: [services]\\n\\nPLAY [Deploy Data Agent] *******************************************************\\n\\nTASK [include soe.yml] *********************************************************\\nok: [services] => (item=/sas/install/ansible/sas_viya_playbook/internal/soe_defaults.yml)\\n\\nTASK [Verify that SAS Data Agent Server config directory exists] ***************\\nok: [services]\\n\\nTASK [Create DataAgent definition file] ****************************************\\nchanged: [services]\\n\\nPLAY [Configure and Deploy Data Agent] *****************************************\\n\\nTASK [include soe.yml] *********************************************************\\nok: [services] => (item=/sas/install/ansible/sas_viya_playbook/internal/soe_defaults.yml)\\n\\nTASK [Verify that SAS Data Agent Server config directory exists] ***************\\nok: [services]\\n\\nTASK [Run data agent framework on x64 redhat hosts] ****************************\\n\\nTASK [dagentsrv on x64_redhat_linux_6-yum - configure-and-start] ***************\\n\\nTASK [dagentsrv on x64_redhat_linux_6-yum - configure.yml] *********************\\n\\nTASK [dagentsrv-x64_redhat_linux_6-yum : include_tasks] ************************\\nincluded: /sas/install/ansible/sas_viya_playbook/roles/framework/tasks/main.yml for services\\n\\nTASK [dagentsrv-x64_redhat_linux_6-yum : include variables for framework action, \\\"certificateAuthority-generateTokenForSecretsAndCertificates-1.1\\\"] ***\\nok: [services]\\n\\nTASK [dagentsrv-x64_redhat_linux_6-yum : On the Ansible controller, create temporary directory] ***\\nchanged: [services]\\n\\nTASK [dagentsrv-x64_redhat_linux_6-yum : Using provider \\\"vault\\\", verify the presence of the desired artifact.] ***\\nok: [services -> services]\\n\\nTASK [dagentsrv-x64_redhat_linux_6-yum : Using provider \\\"vault\\\", create artifact.] ***\\nchanged: [services -> services]\\n\\nTASK [dagentsrv-x64_redhat_linux_6-yum : Using provider \\\"vault\\\", copy artifact to controller.] ***\\nchanged: [services -> services]\\n\\nTASK [dagentsrv-x64_redhat_linux_6-yum : Consumer \\\"dagentsrv\\\" verifies that output directory.] ***\\nchanged: [services]\\n\\nTASK [dagentsrv-x64_redhat_linux_6-yum : Consumer \\\"dagentsrv\\\" copies artifact from controller.] ***\\nchanged: [services]\\n\\nTASK [dagentsrv-x64_redhat_linux_6-yum : On the Ansible controller, clean up temp folder on controller] ***\\nchanged: [services]\\n\\nTASK [dagentsrv-x64_redhat_linux_6-yum : Get stats for /sas/install/ansible/sas_viya_playbook/SASViyaV0300_9C8XK4_Linux_x86-64.txt] ***\\nok: [services]\\n\\nTASK [dagentsrv-x64_redhat_linux_6-yum : Confirm /sas/install/ansible/sas_viya_playbook/SASViyaV0300_9C8XK4_Linux_x86-64.txt exists] ***\\nok: [services]\\n\\nTASK [dagentsrv-x64_redhat_linux_6-yum : Copy from /sas/install/ansible/sas_viya_playbook/SASViyaV0300_9C8XK4_Linux_x86-64.txt to /opt/sas/viya/config/etc/dagentsrv/default/setinit.txt] ***\\nchanged: [services]\\n\\nTASK [dagentsrv on x64_redhat_linux_6-yum - start.yml] *************************\\n\\nTASK [dagentsrv-x64_redhat_linux_6-yum : Ensure sas-viya-dagentsrv-default service is started] ***\\nfatal: [services]: FAILED! => {\\n \\\"changed\\\": false\\n}\\n\\nMSG:\\n\\nUnable to start service sas-viya-dagentsrv-default: Job for sas-viya-dagentsrv-default.service failed because the control process exited with error code. See \\\"systemctl status sas-viya-dagentsrv-default.service\\\" and \\\"journalctl -xe\\\" for details.\\n\\n\\nNO MORE HOSTS LEFT *************************************************************\\n\\nPLAY RECAP *********************************************************************\\ncontroller : ok=196 changed=63 unreachable=0 failed=0 skipped=186 rescued=0 ignored=0 \\nlocalhost : ok=11 changed=1 unreachable=0 failed=0 skipped=7 rescued=0 ignored=0 \\nservices : ok=1891 changed=1132 unreachable=0 failed=1 skipped=1255 rescued=0 ignored=0 \\n\\n848.49user 261.89system 1:22:11elapsed 22%CPU (0avgtext+0avgdata 178308maxresident)k\\n5616inputs+608088outputs (7major+46948992minor)pagefaults 0swaps\\n\\n[stderr]\\n\\\"\\r\\n\\r\\nMore information on troubleshooting is available at https://aka.ms/VMExtensionCSELinuxTroubleshoot \"\r\n }\r\n ]\r\n }\r\n}"}]}

########################################################raw error##########################################################

{
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.",
"details": [
{
"code": "Conflict",
"message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource operation completed with terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"VMExtensionProvisioningError\",\r\n \"message\": \"VM has reported a failure when processing extension 'AnsibleControllerStartup'. Error message: \\\"Enable failed: failed to execute command: command terminated with exit status=2\\n[stdout]\\nendpoint URL=https://services.viya.sas:443/\\n\\nTASK [dagentsrv : Save sas endpoint URL] ***************************************\\nok: [services]\\n\\nPLAY [Deploy Data Agent] *******************************************************\\n\\nTASK [include soe.yml] *********************************************************\\nok: [services] => (item=/sas/install/ansible/sas_viya_playbook/internal/soe_defaults.yml)\\n\\nTASK [Verify that SAS Data Agent Server config directory exists] ***************\\nok: [services]\\n\\nTASK [Create DataAgent definition file] ****************************************\\nchanged: [services]\\n\\nPLAY [Configure and Deploy Data Agent] *****************************************\\n\\nTASK [include soe.yml] *********************************************************\\nok: [services] => (item=/sas/install/ansible/sas_viya_playbook/internal/soe_defaults.yml)\\n\\nTASK [Verify that SAS Data Agent Server config directory exists] ***************\\nok: [services]\\n\\nTASK [Run data agent framework on x64 redhat hosts] ****************************\\n\\nTASK [dagentsrv on x64_redhat_linux_6-yum - configure-and-start] ***************\\n\\nTASK [dagentsrv on x64_redhat_linux_6-yum - configure.yml] *********************\\n\\nTASK [dagentsrv-x64_redhat_linux_6-yum : include_tasks] ************************\\nincluded: /sas/install/ansible/sas_viya_playbook/roles/framework/tasks/main.yml for services\\n\\nTASK [dagentsrv-x64_redhat_linux_6-yum : include variables for framework action, \\\"certificateAuthority-generateTokenForSecretsAndCertificates-1.1\\\"] ***\\nok: [services]\\n\\nTASK [dagentsrv-x64_redhat_linux_6-yum : On the Ansible controller, create temporary directory] ***\\nchanged: [services]\\n\\nTASK [dagentsrv-x64_redhat_linux_6-yum : Using provider \\\"vault\\\", verify the presence of the desired artifact.] ***\\nok: [services -> services]\\n\\nTASK [dagentsrv-x64_redhat_linux_6-yum : Using provider \\\"vault\\\", create artifact.] ***\\nchanged: [services -> services]\\n\\nTASK [dagentsrv-x64_redhat_linux_6-yum : Using provider \\\"vault\\\", copy artifact to controller.] ***\\nchanged: [services -> services]\\n\\nTASK [dagentsrv-x64_redhat_linux_6-yum : Consumer \\\"dagentsrv\\\" verifies that output directory.] ***\\nchanged: [services]\\n\\nTASK [dagentsrv-x64_redhat_linux_6-yum : Consumer \\\"dagentsrv\\\" copies artifact from controller.] ***\\nchanged: [services]\\n\\nTASK [dagentsrv-x64_redhat_linux_6-yum : On the Ansible controller, clean up temp folder on controller] ***\\nchanged: [services]\\n\\nTASK [dagentsrv-x64_redhat_linux_6-yum : Get stats for /sas/install/ansible/sas_viya_playbook/SASViyaV0300_9C8XK4_Linux_x86-64.txt] ***\\nok: [services]\\n\\nTASK [dagentsrv-x64_redhat_linux_6-yum : Confirm /sas/install/ansible/sas_viya_playbook/SASViyaV0300_9C8XK4_Linux_x86-64.txt exists] ***\\nok: [services]\\n\\nTASK [dagentsrv-x64_redhat_linux_6-yum : Copy from /sas/install/ansible/sas_viya_playbook/SASViyaV0300_9C8XK4_Linux_x86-64.txt to /opt/sas/viya/config/etc/dagentsrv/default/setinit.txt] ***\\nchanged: [services]\\n\\nTASK [dagentsrv on x64_redhat_linux_6-yum - start.yml] *************************\\n\\nTASK [dagentsrv-x64_redhat_linux_6-yum : Ensure sas-viya-dagentsrv-default service is started] ***\\nfatal: [services]: FAILED! => {\\n \\\"changed\\\": false\\n}\\n\\nMSG:\\n\\nUnable to start service sas-viya-dagentsrv-default: Job for sas-viya-dagentsrv-default.service failed because the control process exited with error code. See \\\"systemctl status sas-viya-dagentsrv-default.service\\\" and \\\"journalctl -xe\\\" for details.\\n\\n\\nNO MORE HOSTS LEFT *************************************************************\\n\\nPLAY RECAP *********************************************************************\\ncontroller : ok=196 changed=63 unreachable=0 failed=0 skipped=186 rescued=0 ignored=0 \\nlocalhost : ok=11 changed=1 unreachable=0 failed=0 skipped=7 rescued=0 ignored=0 \\nservices : ok=1891 changed=1132 unreachable=0 failed=1 skipped=1255 rescued=0 ignored=0 \\n\\n848.49user 261.89system 1:22:11elapsed 22%CPU (0avgtext+0avgdata 178308maxresident)k\\n5616inputs+608088outputs (7major+46948992minor)pagefaults 0swaps\\n\\n[stderr]\\n\\\"\\r\\n\\r\\nMore information on troubleshooting is available at https://aka.ms/VMExtensionCSELinuxTroubleshoot \"\r\n }\r\n ]\r\n }\r\n}"
}
]
}

Start with the SAS Viya template linked in the issue and trace the AnsibleControllerStartup deployment. Reproduce the failure around the sas-viya-dაგენტsrv-default service, then inspect the suggested systemctl status and journalctl -xe output. Done means the deployment completes without the VM extension failure, with the root cause documented or fixed.

Written by the indexing model from the issue text.