Azure / Azure/azure-linux-extensions

Unable to use Linux Diagnostic Extension without storage account

Open
#1,466 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
333
Forks
278
Avg merge
2d 9h
Merged PRs (30d)
4

Description

Piggybacking on https://github.com/Azure/azure-linux-extensions/issues/944, I saw the fix (https://github.com/Azure/azure-linux-extensions/pull/1010) included a flag to be used in the protected settings when configuring the Linux diagnostic extension ([LAD 4.0](https://docs.microsoft.com/en-us/azure/virtual-machines/extensions/diagnostics-linux)).

Example:
```json
{
"storageAccountName": "xxx",
"storageAccountSasToken": "xxx",
"disableStorageAccount": true,
"sinksConfig": {
"sink": [
{
"name": "MyEventHub",
"type": "EventHub",
"sasURL": "xxx"
}
]
}
}
```

However, the storage account still is required or else the logging to other sinks does not occur. Not only does the storage account (with token) need to be valid, but the virtual machine must have connectivity to the storage account. I tried using a valid storage account to which the VM is not connected through vnet, and see the following repeated in the server's syslog:
```
Dec 3 17:24:29 my-vm systemd[1]: metrics-sourcer.service: Main process exited, code=exited, status=1/FAILURE
Dec 3 17:24:29 my-vm systemd[1]: metrics-sourcer.service: Failed with result 'exit-code'.
Dec 3 17:24:29 my-vm systemd[1]: metrics-sourcer.service: Scheduled restart job, restart counter is at 1.
Dec 3 17:24:29 my-vm systemd[1]: Stopped Custom Modified Telegraf service for Linux Agent metrics sourcing.
Dec 3 17:24:29 my-vm systemd[1]: Started Custom Modified Telegraf service for Linux Agent metrics sourcing.
Dec 3 17:24:29 my-vm telegraf[285732]: 2021-12-03T17:24:29Z I! Starting Telegraf
Dec 3 17:24:44 my-vm systemd[1]: metrics-sourcer.service: Main process exited, code=exited, status=1/FAILURE
Dec 3 17:24:44 my-vm systemd[1]: metrics-sourcer.service: Failed with result 'exit-code'.
Dec 3 17:24:44 my-vm systemd[1]: metrics-sourcer.service: Scheduled restart job, restart counter is at 2.
Dec 3 17:24:44 my-vm systemd[1]: Stopped Custom Modified Telegraf service for Linux Agent metrics sourcing.
```

In the Azure Portal, the extension status shows "Provisioning succeeded", but the status message shows the following message (may be related to https://github.com/Azure/azure-linux-extensions/issues/899?):
> Problem(s) detected in generated mdsd configuration. Can't enable, although this install/enable operation is reported as successful so the VM can complete successful startup. Linux Diagnostic Extension will exit. Config validation message: 2021-12-03T17:24:14.8381650Z: Msgpack array size is not set. Will use default 10485760 items. Msgpack map size is not set. Will use default 10485760 items. Msgpack string size is not set. Will use default 10485760 bytes. Msgpack bin size is not set. Will use default 10485760 bytes. Msgpack ext size is not set. Will use default 10485760 bytes. Msgpack nesting is not set. Will use default 10 levels. 2021-12-03T17:24:14.8382100Z: Event ingestion rate limiting (EPS) is set to 20000 events per second. 2021-12-03T17:24:14.8382560Z: Trying to lock '/var/lib/waagent/Microsoft.Azure.Diagnostics.LinuxDiagnostic-4.0.25.lock', fd=7 2021-12-03T17:24:14.8383130Z: Lock '/var/lib/waagent/Microsoft.Azure.Diagnostics.LinuxDiagnostic-4.0.25.lock' was taken successfully. 2021-12-03T17:24:14.8389740Z: No GCS env var is defined. GCS won't be used. Parse reported these messages: /var/lib/waagent/Microsoft.Azure.Diagnostics.LinuxDiagnostic-4.0.25/xmlCfg.xml(3) Error: Storage credential validation for table storage failed: Forbidden /var/lib/waagent/Microsoft.Azure.Diagnostics.LinuxDiagnostic-4.0.25/xmlCfg.xml(33) Error: No default credentials were defined /var/lib/waagent/Microsoft.Azure.Diagnostics.LinuxDiagnostic-4.0.25/xmlCfg.xml(33) Error: No default credentials were defined /var/lib/waagent/Microsoft.Azure.Diagnostics.LinuxDiagnostic-4.0.25/xmlCfg.xml(33) Error: No default credentials were defined /var/lib/waagent/Microsoft.Azure.Diagnostics.LinuxDiagnostic-4.0.25/xmlCfg.xml(33) Error: No default credentials were defined /var/lib/waagent/Microsoft.Azure.Diagnostics.LinuxDiagnostic-4.0.25/xmlCfg.xml(33) Error: No default credentials were defined /var/lib/waagent/Microsoft.Azure.Diagnostics.LinuxDiagnostic-4.0.25/xmlCfg.xml(33) Error: No default credentials were defined /var/lib/waagent/Microsoft.Azure.Diagnostics.LinuxDiagnostic-4.0.25/xmlCfg.xml(33) Error: No default credentials were defined /var/lib/waagent/Microsoft.Azure.Diagnostics.LinuxDiagnostic-4.0.25/xmlCfg.xml(33) Error: No default credentials were defined /var/lib/waagent/Microsoft.Azure.Diagnostics.LinuxDiagnostic-4.0.25/xmlCfg.xml(33) Error: No default credentials were defined /var/lib/waagent/Microsoft.Azure.Diagnostics.LinuxDiagnostic-4.0.25/xmlCfg.xml(33) Error: No default credentials were defined /var/lib/waagent/Microsoft.Azure.Diagnostics.LinuxDiagnostic-4.0.25/xmlCfg.xml(33) Error: No default credentials were defined /var/lib/waagent/Microsoft.Azure.Diagnostics.LinuxDiagnostic-4.0.25/xmlCfg.xml(33) Error: No default credentials were defined /var/lib/waagent/Microsoft.Azure.Diagnostics.LinuxDiagnostic-4.0.25/xmlCfg.xml(33) Error: No default credentials were defined /var/lib/waagent/Microsoft.Azure.Diagnostics.LinuxDiagnostic-4.0.25/xmlCfg.xml(33) Error: No default credentials were defined

While the `disableStorageAccount` option is nice in that it prevents diagnostics from being duplicated to the storage account, I am opening this ticket to enhance the feature to not need a storage account at all. Doing this would allow for much more customization and flexibility when deploying LAD.

Thank you!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing LAD 4.0 handling of the disableStorageAccount setting and the generated xmlCfg.xml, then reproduce the supplied protected-settings configuration while inspecting metrics-sourcer and Telegraf logs. Done means the extension provisions successfully and sends data to configured non-storage sinks without requiring a valid or reachable storage account.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, linux, python
Domain
cloud, observability-sre
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.