Azure / Azure/azure-linux-extensions
Unable to use Linux Diagnostic Extension without storage account
- Dominant language
- Python
- Stars
- 333
- Forks
- 278
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 4
Description
Piggybacking on https://github.com/Azure/azure-linux-extensions/issues/944, I saw the fix (https://github.com/Azure/azure-linux-extensions/pull/1010) included a flag to be used in the protected settings when configuring the Linux diagnostic extension ([LAD 4.0](https://docs.microsoft.com/en-us/azure/virtual-machines/extensions/diagnostics-linux)).
Example:
```json
{
"storageAccountName": "xxx",
"storageAccountSasToken": "xxx",
"disableStorageAccount": true,
"sinksConfig": {
"sink": [
{
"name": "MyEventHub",
"type": "EventHub",
"sasURL": "xxx"
}
]
}
}
```
However, the storage account still is required or else the logging to other sinks does not occur. Not only does the storage account (with token) need to be valid, but the virtual machine must have connectivity to the storage account. I tried using a valid storage account to which the VM is not connected through vnet, and see the following repeated in the server's syslog:
```
Dec 3 17:24:29 my-vm systemd[1]: metrics-sourcer.service: Main process exited, code=exited, status=1/FAILURE
Dec 3 17:24:29 my-vm systemd[1]: metrics-sourcer.service: Failed with result 'exit-code'.
Dec 3 17:24:29 my-vm systemd[1]: metrics-sourcer.service: Scheduled restart job, restart counter is at 1.
Dec 3 17:24:29 my-vm systemd[1]: Stopped Custom Modified Telegraf service for Linux Agent metrics sourcing.
Dec 3 17:24:29 my-vm systemd[1]: Started Custom Modified Telegraf service for Linux Agent metrics sourcing.
Dec 3 17:24:29 my-vm telegraf[285732]: 2021-12-03T17:24:29Z I! Starting Telegraf
Dec 3 17:24:44 my-vm systemd[1]: metrics-sourcer.service: Main process exited, code=exited, status=1/FAILURE
Dec 3 17:24:44 my-vm systemd[1]: metrics-sourcer.service: Failed with result 'exit-code'.
Dec 3 17:24:44 my-vm systemd[1]: metrics-sourcer.service: Scheduled restart job, restart counter is at 2.
Dec 3 17:24:44 my-vm systemd[1]: Stopped Custom Modified Telegraf service for Linux Agent metrics sourcing.
```
In the Azure Portal, the extension status shows "Provisioning succeeded", but the status message shows the following message (may be related to https://github.com/Azure/azure-linux-extensions/issues/899?):
> Problem(s) detected in generated mdsd configuration. Can't enable, although this install/enable operation is reported as successful so the VM can complete successful startup. Linux Diagnostic Extension will exit. Config validation message: 2021-12-03T17:24:14.8381650Z: Msgpack array size is not set. Will use default 10485760 items. Msgpack map size is not set. Will use default 10485760 items. Msgpack string size is not set. Will use default 10485760 bytes. Msgpack bin size is not set. Will use default 10485760 bytes. Msgpack ext size is not set. Will use default 10485760 bytes. Msgpack nesting is not set. Will use default 10 levels. 2021-12-03T17:24:14.8382100Z: Event ingestion rate limiting (EPS) is set to 20000 events per second. 2021-12-03T17:24:14.8382560Z: Trying to lock '/var/lib/waagent/Microsoft.Azure.Diagnostics.LinuxDiagnostic-4.0.25.lock', fd=7 2021-12-03T17:24:14.8383130Z: Lock '/var/lib/waagent/Microsoft.Azure.Diagnostics.LinuxDiagnostic-4.0.25.lock' was taken successfully. 2021-12-03T17:24:14.8389740Z: No GCS env var is defined. GCS won't be used. Parse reported these messages: /var/lib/waagent/Microsoft.Azure.Diagnostics.LinuxDiagnostic-4.0.25/xmlCfg.xml(3) Error: Storage credential validation for table storage failed: Forbidden /var/lib/waagent/Microsoft.Azure.Diagnostics.LinuxDiagnostic-4.0.25/xmlCfg.xml(33) Error: No default credentials were defined /var/lib/waagent/Microsoft.Azure.Diagnostics.LinuxDiagnostic-4.0.25/xmlCfg.xml(33) Error: No default credentials were defined /var/lib/waagent/Microsoft.Azure.Diagnostics.LinuxDiagnostic-4.0.25/xmlCfg.xml(33) Error: No default credentials were defined /var/lib/waagent/Microsoft.Azure.Diagnostics.LinuxDiagnostic-4.0.25/xmlCfg.xml(33) Error: No default credentials were defined /var/lib/waagent/Microsoft.Azure.Diagnostics.LinuxDiagnostic-4.0.25/xmlCfg.xml(33) Error: No default credentials were defined /var/lib/waagent/Microsoft.Azure.Diagnostics.LinuxDiagnostic-4.0.25/xmlCfg.xml(33) Error: No default credentials were defined /var/lib/waagent/Microsoft.Azure.Diagnostics.LinuxDiagnostic-4.0.25/xmlCfg.xml(33) Error: No default credentials were defined /var/lib/waagent/Microsoft.Azure.Diagnostics.LinuxDiagnostic-4.0.25/xmlCfg.xml(33) Error: No default credentials were defined /var/lib/waagent/Microsoft.Azure.Diagnostics.LinuxDiagnostic-4.0.25/xmlCfg.xml(33) Error: No default credentials were defined /var/lib/waagent/Microsoft.Azure.Diagnostics.LinuxDiagnostic-4.0.25/xmlCfg.xml(33) Error: No default credentials were defined /var/lib/waagent/Microsoft.Azure.Diagnostics.LinuxDiagnostic-4.0.25/xmlCfg.xml(33) Error: No default credentials were defined /var/lib/waagent/Microsoft.Azure.Diagnostics.LinuxDiagnostic-4.0.25/xmlCfg.xml(33) Error: No default credentials were defined /var/lib/waagent/Microsoft.Azure.Diagnostics.LinuxDiagnostic-4.0.25/xmlCfg.xml(33) Error: No default credentials were defined /var/lib/waagent/Microsoft.Azure.Diagnostics.LinuxDiagnostic-4.0.25/xmlCfg.xml(33) Error: No default credentials were defined
While the `disableStorageAccount` option is nice in that it prevents diagnostics from being duplicated to the storage account, I am opening this ticket to enhance the feature to not need a storage account at all. Doing this would allow for much more customization and flexibility when deploying LAD.
Thank you!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing LAD 4.0 handling of the disableStorageAccount setting and the generated xmlCfg.xml, then reproduce the supplied protected-settings configuration while inspecting metrics-sourcer and Telegraf logs. Done means the extension provisions successfully and sends data to configured non-storage sinks without requiring a valid or reachable storage account.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, linux, python
- Domain
- cloud, observability-sre
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100