Azure / Azure/azure-linux-extensions

Huge extension.log files from Microsoft.OSTCExtensions.LinuxDiagnostic | mdsd-lde

Open
#618 19 comments 7 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
333
Forks
278
Avg merge
2d 9h
Merged PRs (30d)
4

Description

On multiple Ubuntu xenial hosts in private-peered Azure, behind a proxy, we are seeing excessive logging to extension.log.

Besides the root cause(s) of that, there seems to be no way to turn off the verbose logging, and there seems to be no related logrotate script.

```
$ sudo find /var/log/ -size +102400k -exec ls -ltrh {} \;
# -rw-r--r-- 1 root root 1.5G Mar 5 03:13 /var/log/azure/Microsoft.OSTCExtensions.LinuxDiagnostic/2.3.9027/extension.log
# -rw-r--r-- 1 root root 16G Jun 26 23:01 /var/log/azure/Microsoft.OSTCExtensions.LinuxDiagnostic/2.3.9029/extension.log
# -rw-r--r-- 1 root root 166M Jun 26 23:00 /var/log/azure/Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux/1.4.60.2/extension.log
```

This is despite:
```
$ grep -i "log" /etc/waagent.conf
# Enable verbose logging (y|n)
Logs.Verbose=n
```

There are a couple of errors in there (but that's not my *primary* point here):
```
$ sudo tail -f /var/log/azure/Microsoft.OSTCExtensions.LinuxDiagnostic/2.3.9029/extension.log
2018/07/19 19:24:52 [Microsoft.OSTCExtensions.LinuxDiagnostic-2.3.9029] RunCmd ps -ef | grep mysql | grep -v grep
2018/07/19 19:24:52 ERROR:CalledProcessError. Error Code is 1
2018/07/19 19:24:52 ERROR:CalledProcessError. Command string was ps -ef | grep mysql | grep -v grep
2018/07/19 19:24:52 ERROR:CalledProcessError. Command result was
2018/07/19 19:24:52 [Microsoft.OSTCExtensions.LinuxDiagnostic-2.3.9029] Return 1:
2018/07/19 19:24:52 [Microsoft.OSTCExtensions.LinuxDiagnostic-2.3.9029] RunCmd ps -ef | grep -E 'httpd|apache2' | grep -v grep
2018/07/19 19:24:52 ERROR:CalledProcessError. Error Code is 1
2018/07/19 19:24:52 ERROR:CalledProcessError. Command string was ps -ef | grep -E 'httpd|apache2' | grep -v grep
2018/07/19 19:24:52 ERROR:CalledProcessError. Command result was
2018/07/19 19:24:52 [Microsoft.OSTCExtensions.LinuxDiagnostic-2.3.9029] Return 1:
2018/07/19 19:24:52 [Microsoft.OSTCExtensions.LinuxDiagnostic-2.3.9029] RunCmd /opt/omi/bin/service_control is-running
2018/07/19 19:24:52 ERROR:CalledProcessError. Error Code is 1
2018/07/19 19:24:52 ERROR:CalledProcessError. Command string was /opt/omi/bin/service_control is-running
2018/07/19 19:24:52 ERROR:CalledProcessError. Command result was
2018/07/19 19:24:52 [Microsoft.OSTCExtensions.LinuxDiagnostic-2.3.9029] Return 1:
2018/07/19 19:24:52 [Microsoft.OSTCExtensions.LinuxDiagnostic-2.3.9029] RunCmd /var/lib/waagent/Microsoft.OSTCExtensions.LinuxDiagnostic-2.3.9029/bin/mdsd -v -c /var/lib/waagent/Microsoft.OSTCExtensions.LinuxDiagnostic-2.3.9029/./xmlCfg.xml
...
2018/07/19 19:27:52 [Microsoft.OSTCExtensions.LinuxDiagnostic-2.3.9029] RunCmd /var/lib/waagent/Microsoft.OSTCExtensions.LinuxDiagnostic-2.3.9029/bin/mdsd -v -c /var/lib/waagent/Microsoft.OSTCExtensions.LinuxDiagnostic-2.3.9029/./xmlCfg.xml
...
2018/07/19 19:28:12 ERROR:CalledProcessError. Error Code is 1
2018/07/19 19:28:12 ERROR:CalledProcessError. Command string was /var/lib/waagent/Microsoft.OSTCExtensions.LinuxDiagnostic-2.3.9029/bin/mdsd -v -c /var/lib/waagent/Microsoft.OSTCExtensions.LinuxDiagnostic-2.3.9029/./xmlCfg.xml
2018/07/19 19:28:12 ERROR:CalledProcessError. Command result was 2018-07-19T19:27:52.4373460Z: Not all GCS env vars are defined. Missing 5: MONITORING_GCS_ENVIRONMENT MONITORING_GCS_ACCOUNT MONITORING_GCS_REGION MONITORING_GCS_CERT_CERTFILE MONITORING_GCS_CERT_KEYFILE. GCS won't be used.
2018/07/19 19:28:12 ERROR:Parse reported these messages:
2018/07/19 19:28:12 ERROR:/var/lib/waagent/Microsoft.OSTCExtensions.LinuxDiagnostic-2.3.9029/./xmlCfg.xml(3) Error: Storage credential validation for table storage failed: Error resolving address
2018/07/19 19:28:12 ERROR:/var/lib/waagent/Microsoft.OSTCExtensions.LinuxDiagnostic-2.3.9029/./xmlCfg.xml(32) Fatal: AddMonikerEventInfo() failed: No default credential is found.
2018/07/19 19:28:12 ERROR:/var/lib/waagent/Microsoft.OSTCExtensions.LinuxDiagnostic-2.3.9029/./xmlCfg.xml(38) Error: No default credentials were defined
2018/07/19 19:28:12 ERROR:/var/lib/waagent/Microsoft.OSTCExtensions.LinuxDiagnostic-2.3.9029/./xmlCfg.xml(43) Error: No default credentials were defined
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.