GoogleCloudPlatform / GoogleCloudPlatform/PerfKitBenchmarker

Spurious error that deletion of OpenStack VM failed

Open
#874 1 comment 0 reactions 1 assignee Claimed by @meteorfox View on GitHub
bug P2
Dominant language
Python
Stars
2k
Forks
562
Avg merge
4h 55m
Merged PRs (30d)
69

Description

In many, but not all, runs of PerfKit Benchmarker on OpenStack, the following error is reported:
`2016-03-10 03:55:46,244 73012e99 Thread-25 speccpu2006(1/1) ERROR Got exception running _DeleteResource: Deletion of DebianBasedOpenStackVirtualMachine failed.`
I think this error is generally accompanied by a later message:
`2016-03-10 03:56:04,206 73012e99 Thread-25 speccpu2006(1/1) INFO Instance not found, may have been already deleted`

The code that spuriously emits this error is as follows. In perfkitbenchmarker/resource.py, `_DeleteResource` contains:

```
self._Delete()
try:
if self._Exists():
raise errors.Resource.RetryableDeletionError(
'Deletion of %s failed.' % type(self).__name__)
```

The problem is that once `self._Delete()` has initiated the deletion of the VM, `self._Exists()` checks on the status of the VM and, depending on timing, could get a result such as this:

```
{"server": {"status": "ACTIVE", "updated": "2016-03-04T10:28:06Z", "hostId": "5949369eedb068d54b06da3abd5b84388d5bd42fcc4606f712752fe5", "addresses": {}, "links": [{"href": "http://X.XX.XX.XXX:8774/v2/4f299816e33f49c7ae3819ee2d7d19d9/servers/a81f5829-af4e-40c7-89ce-fe035bb56c53", "rel": "self"}, {"href": "http://X.XX.XX.XXX:8774/4f299816e33f49c7ae3819ee2d7d19d9/servers/a81f5829-af4e-40c7-89ce-fe035bb56c53", "rel": "bookmark"}], "key_name": "perfkit_key_0_81d13aef", "image": {"id": "5b1d5b74-5f84-4b1d-b43b-d6ab54ba5eb7", "links": [{"href": "http://X.XX.XX.XXX:8774/4f299816e33f49c7ae3819ee2d7d19d9/images/5b1d5b74-5f84-4b1d-b43b-d6ab54ba5eb7", "rel": "bookmark"}]}, "OS-EXT-STS:task_state": "deleting", "OS-EXT-STS:vm_state": "active", "OS-SRV-USG:launched_at": "2016-03-03T23:39:19.000000", "flavor": {"id": "5ffb215e-ad47-40a4-b2ae-cafdc3d517a7", "links": [{"href": "http://X.XX.XX.XXX:8774/4f299816e33f49c7ae3819ee2d7d19d9/flavors/5ffb215e-ad47-40a4-b2ae-cafdc3d517a7", "rel": "bookmark"}]}, "id": "a81f5829-af4e-40c7-89ce-fe035bb56c53", "security_groups": [{"name": "perfkit_sc_group"}], "OS-SRV-USG:terminated_at": null, "OS-EXT-AZ:availability_zone": "nova", "user_id": "69718eb945aa431095556fb221612511", "name": "perfkit_vm_0_81d13aef", "created": "2016-03-03T23:38:59Z", "tenant_id": "4f299816e33f49c7ae3819ee2d7d19d9", "OS-DCF:diskConfig": "MANUAL", "os-extended-volumes:volumes_attached": [{"id": "b40f822a-cf89-40b4-8628-6e4426d2287f"}], "accessIPv4": "", "accessIPv6": "", "progress": 0, "OS-EXT-STS:power_state": 1, "config_drive": "True", "metadata": {}}}
```

(note the "task state" indicated as "deleting") or a result such as this:

```
{"server": {"status": "DELETED", "updated": "2016-03-10T08:55:34Z", "hostId": "c9a3adc9a453678d05fff5d06edc23314319a1c6ef1578c527e9ec76", "addresses": {}, "links": [{"href": "http://X.XX.XX.XXX:8774/v2/4f299816e33f49c7ae3819ee2d7d19d9/servers/3cfa3353-df42-4221-9fb7-01c32be7cd96", "rel": "self"}, {"href": "http://X.XX.XX.XXX:8774/4f299816e33f49c7ae3819ee2d7d19d9/servers/3cfa3353-df42-4221-9fb7-01c32be7cd96", "rel": "bookmark"}], "key_name": "perfkit_key_0_73012e99", "image": {"id": "5b1d5b74-5f84-4b1d-b43b-d6ab54ba5eb7", "links": [{"href": "http://X.XX.1XX.XXX:8774/4f299816e33f49c7ae3819ee2d7d19d9/images/5b1d5b74-5f84-4b1d-b43b-d6ab54ba5eb7", "rel": "bookmark"}]}, "OS-EXT-STS:task_state": null, "OS-EXT-STS:vm_state": "deleted", "OS-SRV-USG:launched_at": "2016-03-09T22:27:57.000000", "flavor": {"id": "5ffb215e-ad47-40a4-b2ae-cafdc3d517a7", "links": [{"href": "http://9.26.19.211:8774/4f299816e33f49c7ae3819ee2d7d19d9/flavors/5ffb215e-ad47-40a4-b2ae-cafdc3d517a7", "rel": "bookmark"}]}, "id": "3cfa3353-df42-4221-9fb7-01c32be7cd96", "security_groups": [{"name": "perfkit_sc_group"}], "OS-SRV-USG:terminated_at": "2016-03-10T08:55:46.000000", "OS-EXT-AZ:availability_zone": "nova", "user_id": "69718eb945aa431095556fb221612511", "name": "perfkit_vm_0_73012e99", "created": "2016-03-09T22:27:36Z", "tenant_id": "4f299816e33f49c7ae3819ee2d7d19d9", "OS-DCF:diskConfig": "MANUAL", "os-extended-volumes:volumes_attached": [{"id": "22cc9c2b-fb1e-4613-ac90-d9e30553ef7f"}], "accessIPv4": "", "accessIPv6": "", "OS-EXT-STS:power_state": 0, "config_drive": "True", "metadata": {}}}
```

(note the "status" indicating "DELETED"). In either case, it is clear that the VM is in the process of being deleted. In either case, the error message is spuriously generated.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.