Issue with extensions when Lib.Dir is modifiend in /etc/waagent.conf

Hi,

As the default value for Lib.Dir variable in /etc/waagent.conf is /var/lib/waagent and /var is mounted with no exec rights in our OS, we decided to modify its location as follows:

`Lib.Dir=/usr/local/waagent`

The issue now is that we can't use extensions because the file /usr/local/waagent/Microsoft.OSTCExtensions.VMAccessForLinux-1.5.16//Utils/constants.py contains the following line:

`LibDir = "/var/lib/waagent"`

Let's note in passing that the default value /var/lib/waagent for LibDir is not a good idea as executable extensions code are placed there. By doing so you create a security hole in a Linux system (and that's why secure Linux systems normally mount /var with noexec rights).

Below the content of the /var/log/azure/Microsoft.OSTCExtensions.VMAccessForLinux/extension.log file:

`2023/10/19 15:49:31 [Microsoft.OSTCExtensions.VMAccessForLinux-1.5.16] sequence number is 0
2023/10/19 15:49:31 [Microsoft.OSTCExtensions.VMAccessForLinux-1.5.16] setting file path is/usr/local/waagent/Microsoft.OSTCExtensions.VMAccessForLinux-1.5.16/config/0.settings
2023/10/19 15:49:31 [Microsoft.OSTCExtensions.VMAccessForLinux-1.5.16] JSON config: {"runtimeSettings": [{"handlerSettings": {"publicSettings": null, "protectedSettings": "*** REDACTED ***", "protectedSettingsCertThumbprint": "*** REDACTED ***"}}]}
2023/10/19 15:49:31 ERROR: CalledProcessError. Error Code is 2
2023/10/19 15:49:31 ERROR: CalledProcessError. Command was ['openssl', 'smime', '-inform', 'DER', '-decrypt', '-recip', '/var/lib/waagent/28FFE9210CC03B8D16BD9A5C29DCD34CD09E825E.crt', '-inkey', '/var/lib/waagent/28FFE9210CC03B8D16BD9A5C29DCD34CD09E825E.prv']
2023/10/19 15:49:31 ERROR: CalledProcessError. Command result was Can't open /var/lib/waagent/28FFE9210CC03B8D16BD9A5C29DCD34CD09E825E.crt for reading, No such file or directory
2023/10/19 15:49:31 ERROR: 139972641794944:error:02001002:system library:fopen:No such file or directory:crypto/bio/bss_file.c:69:fopen('/var/lib/waagent/28FFE9210CC03B8D16BD9A5C29DCD34CD09E825E.crt','r')
2023/10/19 15:49:31 ERROR: 139972641794944:error:2006D080:BIO routines:BIO_new_file:no such file:crypto/bio/bss_file.c:76:
2023/10/19 15:49:31 ERROR: unable to load certificate
2023/10/19 15:49:31 ERROR:
2023/10/19 15:49:31 ERROR: [Microsoft.OSTCExtensions.VMAccessForLinux-1.5.16] JSON exception decoding Can't open /var/lib/waagent/28FFE9210CC03B8D16BD9A5C29DCD34CD09E825E.crt for reading, No such file or directory
2023/10/19 15:49:31 ERROR: 139972641794944:error:02001002:system library:fopen:No such file or directory:crypto/bio/bss_file.c:69:fopen('/var/lib/waagent/28FFE9210CC03B8D16BD9A5C29DCD34CD09E825E.crt','r')
2023/10/19 15:49:31 ERROR: 139972641794944:error:2006D080:BIO routines:BIO_new_file:no such file:crypto/bio/bss_file.c:76:
2023/10/19 15:49:31 ERROR: unable to load certificate
2023/10/19 15:49:31 ERROR:
2023/10/19 15:49:31 [Microsoft.OSTCExtensions.VMAccessForLinux-1.5.16] Config decoded correctly.
2023/10/19 15:49:31 [Microsoft.OSTCExtensions.VMAccessForLinux-1.5.16] Install,success,0,Install Succeeded
2023/10/19 15:49:33 [Microsoft.OSTCExtensions.VMAccessForLinux-1.5.16] sequence number is 0
2023/10/19 15:49:33 [Microsoft.OSTCExtensions.VMAccessForLinux-1.5.16] setting file path is/usr/local/waagent/Microsoft.OSTCExtensions.VMAccessForLinux-1.5.16/config/0.settings
2023/10/19 15:49:33 [Microsoft.OSTCExtensions.VMAccessForLinux-1.5.16] JSON config: {"runtimeSettings": [{"handlerSettings": {"publicSettings": null, "protectedSettings": "*** REDACTED ***", "protectedSettingsCertThumbprint": "*** REDACTED ***"}}]}
2023/10/19 15:49:33 ERROR: CalledProcessError. Error Code is 2
2023/10/19 15:49:33 ERROR: CalledProcessError. Command was ['openssl', 'smime', '-inform', 'DER', '-decrypt', '-recip', '/var/lib/waagent/28FFE9210CC03B8D16BD9A5C29DCD34CD09E825E.crt', '-inkey', '/var/lib/waagent/28FFE9210CC03B8D16BD9A5C29DCD34CD09E825E.prv']
2023/10/19 15:49:33 ERROR: CalledProcessError. Command result was Can't open /var/lib/waagent/28FFE9210CC03B8D16BD9A5C29DCD34CD09E825E.crt for reading, No such file or directory
2023/10/19 15:49:33 ERROR: 140583363873664:error:02001002:system library:fopen:No such file or directory:crypto/bio/bss_file.c:69:fopen('/var/lib/waagent/28FFE9210CC03B8D16BD9A5C29DCD34CD09E825E.crt','r')
2023/10/19 15:49:33 ERROR: 140583363873664:error:2006D080:BIO routines:BIO_new_file:no such file:crypto/bio/bss_file.c:76:
2023/10/19 15:49:33 ERROR: unable to load certificate
2023/10/19 15:49:33 ERROR:
2023/10/19 15:49:33 ERROR: [Microsoft.OSTCExtensions.VMAccessForLinux-1.5.16] JSON exception decoding Can't open /var/lib/waagent/28FFE9210CC03B8D16BD9A5C29DCD34CD09E825E.crt for reading, No such file or directory
2023/10/19 15:49:33 ERROR: 140583363873664:error:02001002:system library:fopen:No such file or directory:crypto/bio/bss_file.c:69:fopen('/var/lib/waagent/28FFE9210CC03B8D16BD9A5C29DCD34CD09E825E.crt','r')
2023/10/19 15:49:33 ERROR: 140583363873664:error:2006D080:BIO routines:BIO_new_file:no such file:crypto/bio/bss_file.c:76:
2023/10/19 15:49:33 ERROR: unable to load certificate
2023/10/19 15:49:33 ERROR:
2023/10/19 15:49:33 [Microsoft.OSTCExtensions.VMAccessForLinux-1.5.16] Config decoded correctly.
2023/10/19 15:49:33 [Microsoft.OSTCExtensions.VMAccessForLinux-1.5.16] set most recent sequence number to 0
2023/10/19 15:49:33 ERROR: Error Traceback (most recent call last):
2023/10/19 15:49:33 ERROR: File "/usr/local/waagent/Microsoft.OSTCExtensions.VMAccessForLinux-1.5.16/Utils/extensionutils.py", line 353, in add_extension_event
2023/10/19 15:49:33 ERROR: event.save()
2023/10/19 15:49:33 ERROR: File "/usr/local/waagent/Microsoft.OSTCExtensions.VMAccessForLinux-1.5.16/Utils/extensionutils.py", line 315, in save
2023/10/19 15:49:33 ERROR: os.mkdir(event_folder)
2023/10/19 15:49:33 ERROR: FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/waagent/events'
2023/10/19 15:49:33 ERROR:
2023/10/19 15:49:33 ERROR: CalledProcessError. Error message is [Errno 2] No such file or directory: '/usr/sbin/service'
2023/10/19 15:49:33 ERROR: Failed to restart SSH service with return code:2
2023/10/19 15:49:33 ERROR: [Microsoft.OSTCExtensions.VMAccessForLinux-1.5.16] Failed to enable the extension with error: 'str' object has no attribute 'get', stack trace: Traceback (most recent call last):
2023/10/19 15:49:33 ERROR: File "/usr/local/waagent/Microsoft.OSTCExtensions.VMAccessForLinux-1.5.16/./vmaccess.py", line 156, in enable
2023/10/19 15:49:33 ERROR: if _is_sshd_config_modified(protect_settings):
2023/10/19 15:49:33 ERROR: File "/usr/local/waagent/Microsoft.OSTCExtensions.VMAccessForLinux-1.5.16/./vmaccess.py", line 201, in _is_sshd_config_modified
2023/10/19 15:49:33 ERROR: result = protected_settings.get('reset_ssh') or protected_settings.get('password')
2023/10/19 15:49:33 ERROR: AttributeError: 'str' object has no attribute 'get'
2023/10/19 15:49:33 ERROR:
2023/10/19 15:49:33 [Microsoft.OSTCExtensions.VMAccessForLinux-1.5.16] Enable,error,0,Enable failed: 'str' object has no attribute 'get'
2023/10/19 15:52:35 [Microsoft.OSTCExtensions.VMAccessForLinux-1.5.16] sequence number is 0
2023/10/19 15:52:35 [Microsoft.OSTCExtensions.VMAccessForLinux-1.5.16] setting file path is/usr/local/waagent/Microsoft.OSTCExtensions.VMAccessForLinux-1.5.16/config/0.settings
2023/10/19 15:52:35 [Microsoft.OSTCExtensions.VMAccessForLinux-1.5.16] JSON config: {"runtimeSettings": [{"handlerSettings": {"publicSettings": null, "protectedSettings": "*** REDACTED ***", "protectedSettingsCertThumbprint": "*** REDACTED ***"}}]}
2023/10/19 15:52:35 ERROR: CalledProcessError. Error Code is 2
2023/10/19 15:52:35 ERROR: CalledProcessError. Command was ['openssl', 'smime', '-inform', 'DER', '-decrypt', '-recip', '/var/lib/waagent/*** REDACTED ***.crt', '-inkey', '/var/lib/waagent/*** REDACTED ***.prv']
2023/10/19 15:52:35 ERROR: CalledProcessError. Command result was Can't open /var/lib/waagent/*** REDACTED ***.crt for reading, No such file or directory
2023/10/19 15:52:35 ERROR: 140075000892288:error:02001002:system library:fopen:No such file or directory:crypto/bio/bss_file.c:69:fopen('/var/lib/waagent/*** REDACTED ***.crt','r')
2023/10/19 15:52:35 ERROR: 140075000892288:error:2006D080:BIO routines:BIO_new_file:no such file:crypto/bio/bss_file.c:76:
2023/10/19 15:52:35 ERROR: unable to load certificate
2023/10/19 15:52:35 ERROR:
2023/10/19 15:52:35 ERROR: [Microsoft.OSTCExtensions.VMAccessForLinux-1.5.16] JSON exception decoding Can't open /var/lib/waagent/*** REDACTED ***.crt for reading, No such file or directory
2023/10/19 15:52:35 ERROR: 140075000892288:error:02001002:system library:fopen:No such file or directory:crypto/bio/bss_file.c:69:fopen('/var/lib/waagent/*** REDACTED ***.crt','r')
2023/10/19 15:52:35 ERROR: 140075000892288:error:2006D080:BIO routines:BIO_new_file:no such file:crypto/bio/bss_file.c:76:
2023/10/19 15:52:35 ERROR: unable to load certificate
2023/10/19 15:52:35 ERROR:
2023/10/19 15:52:35 [Microsoft.OSTCExtensions.VMAccessForLinux-1.5.16] Config decoded correctly.
2023/10/19 15:52:35 [Microsoft.OSTCExtensions.VMAccessForLinux-1.5.16] Current sequence number, 0, is not greater than the sequence number of the most recent executed configuration. Exiting...
2023/10/19 15:52:58 [Microsoft.OSTCExtensions.VMAccessForLinux-1.5.16] sequence number is 0
2023/10/19 15:52:58 [Microsoft.OSTCExtensions.VMAccessForLinux-1.5.16] setting file path is/usr/local/waagent/Microsoft.OSTCExtensions.VMAccessForLinux-1.5.16/config/0.settings
2023/10/19 15:52:58 [Microsoft.OSTCExtensions.VMAccessForLinux-1.5.16] JSON config: {"runtimeSettings": [{"handlerSettings": {"publicSettings": null, "protectedSettings": "*** REDACTED ***", "protectedSettingsCertThumbprint": "*** REDACTED ***"}}]}
2023/10/19 15:52:58 ERROR: CalledProcessError. Error Code is 2
2023/10/19 15:52:58 ERROR: CalledProcessError. Command was ['openssl', 'smime', '-inform', 'DER', '-decrypt', '-recip', '/var/lib/waagent/28FFE9210CC03B8D16BD9A5C29DCD34CD09E825E.crt', '-inkey', '/var/lib/waagent/28FFE9210CC03B8D16BD9A5C29DCD34CD09E825E.prv']
2023/10/19 15:52:58 ERROR: CalledProcessError. Command result was Can't open /var/lib/waagent/28FFE9210CC03B8D16BD9A5C29DCD34CD09E825E.crt for reading, No such file or directory
2023/10/19 15:52:58 ERROR: 140299211979648:error:02001002:system library:fopen:No such file or directory:crypto/bio/bss_file.c:69:fopen('/var/lib/waagent/28FFE9210CC03B8D16BD9A5C29DCD34CD09E825E.crt','r')
2023/10/19 15:52:58 ERROR: 140299211979648:error:2006D080:BIO routines:BIO_new_file:no such file:crypto/bio/bss_file.c:76:
2023/10/19 15:52:58 ERROR: unable to load certificate
2023/10/19 15:52:58 ERROR:
2023/10/19 15:52:58 ERROR: [Microsoft.OSTCExtensions.VMAccessForLinux-1.5.16] JSON exception decoding Can't open /var/lib/waagent/28FFE9210CC03B8D16BD9A5C29DCD34CD09E825E.crt for reading, No such file or directory
2023/10/19 15:52:58 ERROR: 140299211979648:error:02001002:system library:fopen:No such file or directory:crypto/bio/bss_file.c:69:fopen('/var/lib/waagent/28FFE9210CC03B8D16BD9A5C29DCD34CD09E825E.crt','r')
2023/10/19 15:52:58 ERROR: 140299211979648:error:2006D080:BIO routines:BIO_new_file:no such file:crypto/bio/bss_file.c:76:
2023/10/19 15:52:58 ERROR: unable to load certificate
2023/10/19 15:52:58 ERROR:
2023/10/19 15:52:58 [Microsoft.OSTCExtensions.VMAccessForLinux-1.5.16] Config decoded correctly.
2023/10/19 15:52:58 [Microsoft.OSTCExtensions.VMAccessForLinux-1.5.16] Current sequence number, 0, is not greater than the sequence number of the most recent executed configuration. Exiting...
`

As you can see extensions codes can't even find the certificate...

Is there a way to configure the agent to place extensions in another location? Or modify the Lib.Dir value in /usr/local/waagent/Microsoft.OSTCExtensions.VMAccessForLinux-1.5.16//Utils/constants.py during the initialization phase?

Best Regards,
The CacheGuard Dev Team

Start with /etc/waagent.conf and the hard-coded LibDir in the extension's Utils/constants.py, then trace the certificate and event paths through Utils/extensionutils.py and vmaccess.py. Confirm how a non-default Lib.Dir should propagate to extensions and verify that certificate lookup and event creation use the configured location instead of /var/lib/waagent.

Written by the indexing model from the issue text.