Azure / Azure/WALinuxAgent

[BUG] /var/lib/waagent/history taking up disk space

Open
#2,836 14 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
581
Forks
397
Avg merge
2d 15h
Merged PRs (30d)
9

Description

/var/lib/waagent/history takes up disk space. It contains 1263464 files taking up 20GB.

```sh
# ls -1 /var/lib/waagent/history | wc -l
1263464
# du -sh /var/lib/waagent/history 2>/dev/null
20G /var/lib/waagent/history
```

**Distro and WALinuxAgent details (please complete the following information):**
- Distro and Version: Ubuntu 22.04
- WALinuxAgent version
```sh
/usr/sbin/waagent:27: DeprecationWarning: the imp module is deprecated in favour of importlib and slated for removal in Python 3.12; see the module's documentation for alternative uses
import imp
WALinuxAgent-2.2.46 running on ubuntu 22.04
Python: 3.10.6
Goal state agent: 2.2.46
```

**Log file attached**
If possible, please provide the full /var/log/waagent.log file to help us understand the problem better and get the context of the issue.
The log is full of the same lines :

```
2023-06-05T16:18:26.461436Z INFO ExtHandler WireServer is not responding. Reset dhcp endpoint
2023-06-05T16:18:26.467519Z INFO ExtHandler Protocol endpoint not found: WireProtocol, [ProtocolError] Exceeded max retry updating goal state
2023-06-05T16:18:26.479238Z INFO ExtHandler Protocol endpoint not found: MetadataProtocol, [ProtocolError] 404 - GET: http://169.254.169.254/Microsoft.Compute/identity?api-version=2015-05-01-preview
2023-06-05T16:18:26.491881Z INFO ExtHandler Retry detect protocols: retry=258
2023-06-05T16:18:36.506814Z INFO ExtHandler WireServer endpoint is not found. Rerun dhcp handler
2023-06-05T16:18:36.513326Z INFO ExtHandler Test for route to 168.63.129.16
2023-06-05T16:18:36.518117Z INFO ExtHandler Route to 168.63.129.16 exists
2023-06-05T16:18:36.522790Z INFO ExtHandler Wire server endpoint:168.63.129.16
2023-06-05T16:18:36.530545Z INFO ExtHandler Fabric preferred wire protocol version:2015-04-05
2023-06-05T16:18:36.536644Z INFO ExtHandler Wire protocol version:2012-11-30
2023-06-05T16:18:36.541396Z INFO ExtHandler Server preferred version:2015-04-05
2023-06-05T16:18:36.905773Z INFO ExtHandler WireServer is not responding. Reset dhcp endpoint
2023-06-05T16:18:36.911830Z INFO ExtHandler Protocol endpoint not found: WireProtocol, [ProtocolError] Exceeded max retry updating goal state
2023-06-05T16:18:36.923324Z INFO ExtHandler Protocol endpoint not found: MetadataProtocol, [ProtocolError] 404 - GET: http://169.254.169.254/Microsoft.Compute/identity?api-version=2015-05-01-preview
2023-06-05T16:18:36.935750Z INFO ExtHandler Retry detect protocols: retry=259
2023-06-05T16:18:46.950708Z INFO ExtHandler WireServer endpoint is not found. Rerun dhcp handler
2023-06-05T16:18:46.957058Z INFO ExtHandler Test for route to 168.63.129.16
2023-06-05T16:18:46.961926Z INFO ExtHandler Route to 168.63.129.16 exists
2023-06-05T16:18:46.966761Z INFO ExtHandler Wire server endpoint:168.63.129.16
2023-06-05T16:18:46.974642Z INFO ExtHandler Fabric preferred wire protocol version:2015-04-05
2023-06-05T16:18:46.980745Z INFO ExtHandler Wire protocol version:2012-11-30
2023-06-05T16:18:46.985510Z INFO ExtHandler Server preferred version:2015-04-05
2023-06-05T16:18:47.391438Z INFO ExtHandler WireServer is not responding. Reset dhcp endpoint
2023-06-05T16:18:47.397516Z INFO ExtHandler Protocol endpoint not found: WireProtocol, [ProtocolError] Exceeded max retry updating goal state
2023-06-05T16:18:47.409258Z INFO ExtHandler Protocol endpoint not found: MetadataProtocol, [ProtocolError] 404 - GET: http://169.254.169.254/Microsoft.Compute/identity?api-version=2015-05-01-preview
2023-06-05T16:18:47.421506Z INFO ExtHandler Retry detect protocols: retry=260
2023-06-05T16:18:57.436627Z INFO ExtHandler WireServer endpoint is not found. Rerun dhcp handler
2023-06-05T16:18:57.443070Z INFO ExtHandler Test for route to 168.63.129.16
2023-06-05T16:18:57.448002Z INFO ExtHandler Route to 168.63.129.16 exists
2023-06-05T16:18:57.452518Z INFO ExtHandler Wire server endpoint:168.63.129.16
2023-06-05T16:18:57.460228Z INFO ExtHandler Fabric preferred wire protocol version:2015-04-05
2023-06-05T16:18:57.466212Z INFO ExtHandler Wire protocol version:2012-11-30
2023-06-05T16:18:57.471221Z INFO ExtHandler Server preferred version:2015-04-05
2023-06-05T16:18:57.784699Z INFO ExtHandler WireServer is not responding. Reset dhcp endpoint
2023-06-05T16:18:57.790682Z INFO ExtHandler Protocol endpoint not found: WireProtocol, [ProtocolError] Exceeded max retry updating goal state
2023-06-05T16:18:57.801999Z INFO ExtHandler Protocol endpoint not found: MetadataProtocol, [ProtocolError] 404 - GET: http://169.254.169.254/Microsoft.Compute/identity?api-version=2015-05-01-preview
2023-06-05T16:18:57.814302Z INFO ExtHandler Retry detect protocols: retry=261
2023-06-05T16:19:07.827648Z INFO ExtHandler WireServer endpoint is not found. Rerun dhcp handler
2023-06-05T16:19:07.834202Z INFO ExtHandler Test for route to 168.63.129.16
2023-06-05T16:19:07.839029Z INFO ExtHandler Route to 168.63.129.16 exists
2023-06-05T16:19:07.843738Z INFO ExtHandler Wire server endpoint:168.63.129.16
2023-06-05T16:19:07.851647Z INFO ExtHandler Fabric preferred wire protocol version:2015-04-05
2023-06-05T16:19:07.857456Z INFO ExtHandler Wire protocol version:2012-11-30
2023-06-05T16:19:07.862201Z INFO ExtHandler Server preferred version:2015-04-05
2023-06-05T16:19:08.385468Z INFO ExtHandler WireServer is not responding. Reset dhcp endpoint
2023-06-05T16:19:08.391785Z INFO ExtHandler Protocol endpoint not found: WireProtocol, [ProtocolError] Exceeded max retry updating goal state
2023-06-05T16:19:08.403696Z INFO ExtHandler Protocol endpoint not found: MetadataProtocol, [ProtocolError] 404 - GET: http://169.254.169.254/Microsoft.Compute/identity?api-version=2015-05-01-preview
2023-06-05T16:19:08.416142Z INFO ExtHandler Retry detect protocols: retry=262
2023-06-05T16:19:18.431431Z INFO ExtHandler WireServer endpoint is not found. Rerun dhcp handler
2023-06-05T16:19:18.437793Z INFO ExtHandler Test for route to 168.63.129.16
2023-06-05T16:19:18.442681Z INFO ExtHandler Route to 168.63.129.16 exists
2023-06-05T16:19:18.447416Z INFO ExtHandler Wire server endpoint:168.63.129.16
2023-06-05T16:19:18.455172Z INFO ExtHandler Fabric preferred wire protocol version:2015-04-05
2023-06-05T16:19:18.461181Z INFO ExtHandler Wire protocol version:2012-11-30
2023-06-05T16:19:18.465981Z INFO ExtHandler Server preferred version:2015-04-05
2023-06-05T16:19:18.931679Z INFO ExtHandler WireServer is not responding. Reset dhcp endpoint
2023-06-05T16:19:18.937940Z INFO ExtHandler Protocol endpoint not found: WireProtocol, [ProtocolError] Exceeded max retry updating goal state
2023-06-05T16:19:18.949539Z INFO ExtHandler Protocol endpoint not found: MetadataProtocol, [ProtocolError] 404 - GET: http://169.254.169.254/Microsoft.Compute/identity?api-version=2015-05-01-preview
2023-06-05T16:19:18.961922Z INFO ExtHandler Retry detect protocols: retry=263
2023-06-05T16:19:28.967774Z INFO ExtHandler WireServer endpoint is not found. Rerun dhcp handler
2023-06-05T16:19:28.974222Z INFO ExtHandler Test for route to 168.63.129.16
2023-06-05T16:19:28.979193Z INFO ExtHandler Route to 168.63.129.16 exists
2023-06-05T16:19:28.983903Z INFO ExtHandler Wire server endpoint:168.63.129.16
2023-06-05T16:19:28.991942Z INFO ExtHandler Fabric preferred wire protocol version:2015-04-05
2023-06-05T16:19:28.997854Z INFO ExtHandler Wire protocol version:2012-11-30
2023-06-05T16:19:29.002804Z INFO ExtHandler Server preferred version:2015-04-05
2023-06-05T16:19:29.509588Z INFO ExtHandler WireServer is not responding. Reset dhcp endpoint
2023-06-05T16:19:29.515638Z INFO ExtHandler Protocol endpoint not found: WireProtocol, [ProtocolError] Exceeded max retry updating goal state
2023-06-05T16:19:29.527222Z INFO ExtHandler Protocol endpoint not found: MetadataProtocol, [ProtocolError] 404 - GET: http://169.254.169.254/Microsoft.Compute/identity?api-version=2015-05-01-preview
2023-06-05T16:19:29.539680Z INFO ExtHandler Retry detect protocols: retry=264
2023-06-05T16:19:39.551651Z INFO ExtHandler WireServer endpoint is not found. Rerun dhcp handler
2023-06-05T16:19:39.558109Z INFO ExtHandler Test for route to 168.63.129.16
2023-06-05T16:19:39.563153Z INFO ExtHandler Route to 168.63.129.16 exists
2023-06-05T16:19:39.567919Z INFO ExtHandler Wire server endpoint:168.63.129.16
2023-06-05T16:19:39.576132Z INFO ExtHandler Fabric preferred wire protocol version:2015-04-05
2023-06-05T16:19:39.582168Z INFO ExtHandler Wire protocol version:2012-11-30
2023-06-05T16:19:39.587064Z INFO ExtHandler Server preferred version:2015-04-05
2023-06-05T16:19:40.135079Z INFO ExtHandler WireServer is not responding. Reset dhcp endpoint
2023-06-05T16:19:40.141299Z INFO ExtHandler Protocol endpoint not found: WireProtocol, [ProtocolError] Exceeded max retry updating goal state
2023-06-05T16:19:40.153113Z INFO ExtHandler Protocol endpoint not found: MetadataProtocol, [ProtocolError] 404 - GET: http://169.254.169.254/Microsoft.Compute/identity?api-version=2015-05-01-preview
2023-06-05T16:19:40.165547Z INFO ExtHandler Retry detect protocols: retry=265
2023-06-05T16:19:50.180569Z INFO ExtHandler WireServer endpoint is not found. Rerun dhcp handler
2023-06-05T16:19:50.187195Z INFO ExtHandler Test for route to 168.63.129.16
2023-06-05T16:19:50.192353Z INFO ExtHandler Route to 168.63.129.16 exists
2023-06-05T16:19:50.197232Z INFO ExtHandler Wire server endpoint:168.63.129.16
2023-06-05T16:19:50.205120Z INFO ExtHandler Fabric preferred wire protocol version:2015-04-05
2023-06-05T16:19:50.211259Z INFO ExtHandler Wire protocol version:2012-11-30
2023-06-05T16:19:50.216139Z INFO ExtHandler Server preferred version:2015-04-05
2023-06-05T16:19:50.671650Z INFO ExtHandler WireServer is not responding. Reset dhcp endpoint
2023-06-05T16:19:50.677806Z INFO ExtHandler Protocol endpoint not found: WireProtocol, [ProtocolError] Exceeded max retry updating goal state
2023-06-05T16:19:50.689646Z INFO ExtHandler Protocol endpoint not found: MetadataProtocol, [ProtocolError] 404 - GET: http://169.254.169.254/Microsoft.Compute/identity?api-version=2015-05-01-preview
2023-06-05T16:19:50.702160Z INFO ExtHandler Retry detect protocols: retry=266
2023-06-05T16:20:00.711736Z INFO ExtHandler WireServer endpoint is not found. Rerun dhcp handler
2023-06-05T16:20:00.718395Z INFO ExtHandler Test for route to 168.63.129.16
2023-06-05T16:20:00.723445Z INFO ExtHandler Route to 168.63.129.16 exists
2023-06-05T16:20:00.728345Z INFO ExtHandler Wire server endpoint:168.63.129.16
2023-06-05T16:20:00.736203Z INFO ExtHandler Fabric preferred wire protocol version:2015-04-05
2023-06-05T16:20:00.742465Z INFO ExtHandler Wire protocol version:2012-11-30
2023-06-05T16:20:00.747684Z INFO ExtHandler Server preferred version:2015-04-05
```

Contributor guide

Open the contributing guide

Research direction

Start by examining /var/lib/waagent/history and the repeated failures in /var/log/waagent.log, then trace how the WALinuxAgent creates history files during these retries. Done means the accumulation cause is understood and a bounded history or cleanup behavior is defined and verified.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, python
Domain
cloud, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.