GoogleCloudPlatform / GoogleCloudPlatform/PerfKitBenchmarker
Spurious error that deletion of OpenStack VM failed
- Dominant language
- Python
- Stars
- 2k
- Forks
- 562
- Avg merge
- 4h 55m
- Merged PRs (30d)
- 69
Description
In many, but not all, runs of PerfKit Benchmarker on OpenStack, the following error is reported:
`2016-03-10 03:55:46,244 73012e99 Thread-25 speccpu2006(1/1) ERROR Got exception running _DeleteResource: Deletion of DebianBasedOpenStackVirtualMachine failed.`
I think this error is generally accompanied by a later message:
`2016-03-10 03:56:04,206 73012e99 Thread-25 speccpu2006(1/1) INFO Instance not found, may have been already deleted`
The code that spuriously emits this error is as follows. In perfkitbenchmarker/resource.py, `_DeleteResource` contains:
```
self._Delete()
try:
if self._Exists():
raise errors.Resource.RetryableDeletionError(
'Deletion of %s failed.' % type(self).__name__)
```
The problem is that once `self._Delete()` has initiated the deletion of the VM, `self._Exists()` checks on the status of the VM and, depending on timing, could get a result such as this:
```
{"server": {"status": "ACTIVE", "updated": "2016-03-04T10:28:06Z", "hostId": "5949369eedb068d54b06da3abd5b84388d5bd42fcc4606f712752fe5", "addresses": {}, "links": [{"href": "http://X.XX.XX.XXX:8774/v2/4f299816e33f49c7ae3819ee2d7d19d9/servers/a81f5829-af4e-40c7-89ce-fe035bb56c53", "rel": "self"}, {"href": "http://X.XX.XX.XXX:8774/4f299816e33f49c7ae3819ee2d7d19d9/servers/a81f5829-af4e-40c7-89ce-fe035bb56c53", "rel": "bookmark"}], "key_name": "perfkit_key_0_81d13aef", "image": {"id": "5b1d5b74-5f84-4b1d-b43b-d6ab54ba5eb7", "links": [{"href": "http://X.XX.XX.XXX:8774/4f299816e33f49c7ae3819ee2d7d19d9/images/5b1d5b74-5f84-4b1d-b43b-d6ab54ba5eb7", "rel": "bookmark"}]}, "OS-EXT-STS:task_state": "deleting", "OS-EXT-STS:vm_state": "active", "OS-SRV-USG:launched_at": "2016-03-03T23:39:19.000000", "flavor": {"id": "5ffb215e-ad47-40a4-b2ae-cafdc3d517a7", "links": [{"href": "http://X.XX.XX.XXX:8774/4f299816e33f49c7ae3819ee2d7d19d9/flavors/5ffb215e-ad47-40a4-b2ae-cafdc3d517a7", "rel": "bookmark"}]}, "id": "a81f5829-af4e-40c7-89ce-fe035bb56c53", "security_groups": [{"name": "perfkit_sc_group"}], "OS-SRV-USG:terminated_at": null, "OS-EXT-AZ:availability_zone": "nova", "user_id": "69718eb945aa431095556fb221612511", "name": "perfkit_vm_0_81d13aef", "created": "2016-03-03T23:38:59Z", "tenant_id": "4f299816e33f49c7ae3819ee2d7d19d9", "OS-DCF:diskConfig": "MANUAL", "os-extended-volumes:volumes_attached": [{"id": "b40f822a-cf89-40b4-8628-6e4426d2287f"}], "accessIPv4": "", "accessIPv6": "", "progress": 0, "OS-EXT-STS:power_state": 1, "config_drive": "True", "metadata": {}}}
```
(note the "task state" indicated as "deleting") or a result such as this:
```
{"server": {"status": "DELETED", "updated": "2016-03-10T08:55:34Z", "hostId": "c9a3adc9a453678d05fff5d06edc23314319a1c6ef1578c527e9ec76", "addresses": {}, "links": [{"href": "http://X.XX.XX.XXX:8774/v2/4f299816e33f49c7ae3819ee2d7d19d9/servers/3cfa3353-df42-4221-9fb7-01c32be7cd96", "rel": "self"}, {"href": "http://X.XX.XX.XXX:8774/4f299816e33f49c7ae3819ee2d7d19d9/servers/3cfa3353-df42-4221-9fb7-01c32be7cd96", "rel": "bookmark"}], "key_name": "perfkit_key_0_73012e99", "image": {"id": "5b1d5b74-5f84-4b1d-b43b-d6ab54ba5eb7", "links": [{"href": "http://X.XX.1XX.XXX:8774/4f299816e33f49c7ae3819ee2d7d19d9/images/5b1d5b74-5f84-4b1d-b43b-d6ab54ba5eb7", "rel": "bookmark"}]}, "OS-EXT-STS:task_state": null, "OS-EXT-STS:vm_state": "deleted", "OS-SRV-USG:launched_at": "2016-03-09T22:27:57.000000", "flavor": {"id": "5ffb215e-ad47-40a4-b2ae-cafdc3d517a7", "links": [{"href": "http://9.26.19.211:8774/4f299816e33f49c7ae3819ee2d7d19d9/flavors/5ffb215e-ad47-40a4-b2ae-cafdc3d517a7", "rel": "bookmark"}]}, "id": "3cfa3353-df42-4221-9fb7-01c32be7cd96", "security_groups": [{"name": "perfkit_sc_group"}], "OS-SRV-USG:terminated_at": "2016-03-10T08:55:46.000000", "OS-EXT-AZ:availability_zone": "nova", "user_id": "69718eb945aa431095556fb221612511", "name": "perfkit_vm_0_73012e99", "created": "2016-03-09T22:27:36Z", "tenant_id": "4f299816e33f49c7ae3819ee2d7d19d9", "OS-DCF:diskConfig": "MANUAL", "os-extended-volumes:volumes_attached": [{"id": "22cc9c2b-fb1e-4613-ac90-d9e30553ef7f"}], "accessIPv4": "", "accessIPv6": "", "OS-EXT-STS:power_state": 0, "config_drive": "True", "metadata": {}}}
```
(note the "status" indicating "DELETED"). In either case, it is clear that the VM is in the process of being deleted. In either case, the error message is spuriously generated.
Contributor guide
Assessment
This issue has not been assessed yet.