canonical / canonical/cloud-init
Occasionally fail to retrieve metadata
- Dominant language
- Python
- Stars
- 3.8k
- Forks
- 1.1k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 18
Description
# Bug report
Occasionally fail to retrieve metadata on OpenStack, please see env details.
I have already obtain the dhcp release, but timeout to request the target 169.254.169.254 and raise "No active metadata service found"
and then when the vm startup, I can get the 169.254.169.254.
Did the vm as a client startup was not completely? Please help me how can I confirm that, thanks. My vm network is:
root@lbq111:~# ip a
1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp1s0: mtu 1500 qdisc mq state UP group default qlen 1000
link/ether fa:16:3e:b6:91:5c brd ff:ff:ff:ff:ff:ff
inet 192.168.0.94/24 metric 100 brd 192.168.0.255 scope global enp1s0
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:feb6:915c/64 scope link
valid_lft forever preferred_lft forever
## Steps to reproduce the problem
Occasionally, no fixed steps.
## Environment details
- Cloud-init version: 24.3.1
- Operating System Distribution: VM os is Ubuntu
- Cloud provider, platform or installer type: OpenStack
## cloud-init logs
2025-05-29 02:45:01,000 - ephemeral.py[DEBUG]: Received dhcp lease on enp1s0 for 192.168.0.94/255.255.255.0
2025-05-29 02:45:01,000 - ephemeral.py[DEBUG]: Attempting setup of ephemeral network on enp1s0 with 192.168.0.94/24 brd 192.168.0.255
2025-05-29 02:45:01,001 - subp.py[DEBUG]: Running command ['ip', '--json', 'addr'] with allowed return codes [0] (shell=False, capture=True)
2025-05-29 02:45:01,003 - subp.py[DEBUG]: Running command ['ip', '-family', 'inet', 'addr', 'add', '192.168.0.94/24', 'broadcast', '192.168.0.255', 'dev', 'enp1s0'] with allowed return codes [0] (shell=False, capture=True)
2025-05-29 02:45:01,006 - ephemeral.py[DEBUG]: Skip bringing up network link: interface enp1s0 is already up
2025-05-29 02:45:01,006 - subp.py[DEBUG]: Running command ['ip', '-4', 'route', 'append', '169.254.169.254/32', 'via', '192.168.0.2', 'dev', 'enp1s0'] with allowed return codes [0] (shell=False, capture=True)
2025-05-29 02:45:01,010 - subp.py[DEBUG]: Running command ['ip', '-4', 'route', 'append', '0.0.0.0/0', 'via', '192.168.0.1', 'dev', 'enp1s0'] with allowed return codes [0] (shell=False, capture=True)
2025-05-29 02:45:01,014 - util.py[DEBUG]: Resolving URL: http://[fe80::a9fe:a9fe%25enp1s0] took 0.002 seconds
2025-05-29 02:45:01,014 - util.py[DEBUG]: Resolving URL: http://169.254.169.254 took 0.000 seconds
2025-05-29 02:45:01,016 - url_helper.py[DEBUG]: [0/1] open 'http://[fe80::a9fe:a9fe%25enp1s0]/openstack' with {'url': 'http://[fe80::a9fe:a9fe%25enp1s0]/openstack', 'stream': False, 'allow_redirects': True, 'method': 'GET', 'timeout': 10.0, 'headers': {'User-Agent': 'Cloud-Init/24.3.1-0ubuntu0~22.04.1'}} configuration
2025-05-29 02:45:01,168 - url_helper.py[DEBUG]: [0/1] open 'http://169.254.169.254/openstack' with {'url': 'http://169.254.169.254/openstack', 'stream': False, 'allow_redirects': True, 'method': 'GET', 'timeout': 10.0, 'headers': {'User-Agent': 'Cloud-Init/24.3.1-0ubuntu0~22.04.1'}} configuration
2025-05-29 02:45:11,018 - url_helper.py[WARNING]: Timed out waiting for addresses: http://[fe80::a9fe:a9fe%25enp1s0]/openstack http://169.254.169.254/openstack, exception(s) raised while waiting: HTTPConnectionPool(host='fe80::a9fe:a9fe%enp1s0', port=80): Max retries exceeded with url: /openstack (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 99] Cannot assign requested address'))
2025-05-29 02:45:11,018 - url_helper.py[DEBUG]: Calling 'http://[fe80::a9fe:a9fe%25enp1s0]/openstack' failed [10/-1s]: unexpected error ['NoneType' object has no attribute 'contents']
2025-05-29 02:45:11,018 - url_helper.py[ERROR]: Timed out, no response from urls: ['http://[fe80::a9fe:a9fe%25enp1s0]/openstack', 'http://169.254.169.254/openstack']
2025-05-29 02:45:11,019 - DataSourceOpenStack.py[DEBUG]: Giving up on OpenStack md from ['http://[fe80::a9fe:a9fe%25enp1s0]/openstack', 'http://169.254.169.254/openstack'] after 10 seconds
2025-05-29 02:45:11,019 - util.py[DEBUG]: Crawl of metadata service took 10.007 seconds
2025-05-29 02:45:11,019 - subp.py[DEBUG]: Running command ['ip', '-4', 'route', 'del', '0.0.0.0/0', 'via', '192.168.0.1', 'dev', 'enp1s0'] with allowed return codes [0] (shell=False, capture=True)
2025-05-29 02:45:11,021 - subp.py[DEBUG]: Running command ['ip', '-4', 'route', 'del', '169.254.169.254/32', 'via', '192.168.0.2', 'dev', 'enp1s0'] with allowed return codes [0] (shell=False, capture=True)
2025-05-29 02:45:11,025 - subp.py[DEBUG]: Running command ['ip', '-family', 'inet', 'link', 'set', 'dev', 'enp1s0', 'down'] with allowed return codes [0] (shell=False, capture=True)
2025-05-29 02:45:11,027 - subp.py[DEBUG]: Running command ['ip', '-family', 'inet', 'addr', 'del', '192.168.0.94/24', 'dev', 'enp1s0'] with allowed return codes [0] (shell=False, capture=True)
2025-05-29 02:45:11,030 - util.py[WARNING]: No active metadata service found
2025-05-29 02:45:11,030 - util.py[DEBUG]: No active metadata service found
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/cloudinit/sources/DataSourceOpenStack.py", line 164, in _get_data
results = util.log_time(
File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 2863, in log_time
ret = func(*args, **kwargs)
File "/usr/lib/python3/dist-packages/cloudinit/sources/DataSourceOpenStack.py", line 216, in _crawl_metadata
raise sources.InvalidMetaDataException(
cloudinit.sources.InvalidMetaDataException: No active metadata service found
2025-05-29 02:45:11,035 - sources[DEBUG]: Datasource DataSourceOpenStackLocal [net,ver=None] not updated for events: boot-new-instance
2025-05-29 02:45:11,035 - handlers.py[DEBUG]: finish: init-local/search-OpenStackLocal: SUCCESS: no local data found from DataSourceOpenStackLocal
2025-05-29 02:45:11,035 - util.py[DEBUG]: Reading from /var/lib/cloud/instance/obj.pkl (quiet=False)
2025-05-29 02:45:11,035 - util.py[DEBUG]: Read 11113 bytes from /var/lib/cloud/instance/obj.pkl
2025-05-29 02:45:11,036 - util.py[DEBUG]: Attempting to remove /var/lib/cloud/instance
2025-05-29 02:45:11,037 - main.py[DEBUG]: No local datasource found
2025-05-29 02:45:11,038 - util.py[DEBUG]: Reading from /sys/class/net/lo/address (quiet=False)
Contributor guide
Assessment
This issue has not been assessed yet.