GoogleCloudPlatform / GoogleCloudPlatform/PerfKitBenchmarker

object_storage_service fails during cleanup - tries to delete instance, firewall and network multiple times, after successful STDERR of it being deleted

Open
#1,239 11 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2k
Forks
562
Avg merge
4h 55m
Merged PRs (30d)
69

Description

Hi,

I am unable to get the `object_storage_service` benchmark to work besides when using the `--object_storage_scenario=api_multistream` flag . All others,`cli, api_data, api_namespace` don't work and fail during cleanup.

What seems to be happening is that `pkb` is deleting the instance, `STDERR` writes that it is successfully deleted, then `pkb` tries to delete the now terminated instance, thus failing. I have run into this at least a half dozen times the last two days, and it just clicked that all the log files all show it failing at the instance deletion, and then the loop continues with `firewall` and `network` resources too. Since I don't know python at all, I wouldn't even know where to start digging into `vm_util.py` to see where things might be going awry. If, even, `vm_util.py` is where the bug lies.

```
2016-12-08 00:42:09,113 a6869ecd Thread-469 object_storage_service(1/1) vm_util.py:284 DEBUG Environment variables: None
2016-12-08 00:42:09,113 a6869ecd Thread-469 object_storage_service(1/1) vm_util.py:287 INFO Running: gcloud compute instances delete pkb-a6869ecd-0 --format json --quiet --project REDACTED --zone us-central1-a
2016-12-08 00:42:56,859 a6869ecd Thread-469 object_storage_service(1/1) vm_util.py:318 DEBUG Ran gcloud compute instances delete pkb-a6869ecd-0 --format json --quiet --project REDACTED --zone us-central1-a. Got return code (0).
STDOUT: []

STDERR: Deleted [https://www.googleapis.com/compute/v1/projects/REDACTED/zones/us-central1-a/instances/pkb-a6869ecd-0].

2016-12-08 00:42:56,860 a6869ecd Thread-469 object_storage_service(1/1) vm_util.py:284 DEBUG Environment variables: None
2016-12-08 00:42:56,860 a6869ecd Thread-469 object_storage_service(1/1) vm_util.py:287 INFO Running: gcloud compute instances describe pkb-a6869ecd-0 --format json --quiet --project REDACTED --zone us-central1-a
2016-12-08 00:42:57,473 a6869ecd Thread-469 object_storage_service(1/1) vm_util.py:318 DEBUG Ran gcloud compute instances describe pkb-a6869ecd-0 --format json --quiet --project REDACTED --zone us-central1-a. Got return code (1).
STDOUT:
STDERR: ERROR: (gcloud.compute.instances.describe) Could not fetch resource:
- The resource 'projects/REDACTED/zones/us-central1-a/instances/pkb-a6869ecd-0' was not found

2016-12-08 00:42:57,499 a6869ecd MainThread object_storage_service(1/1) vm_util.py:284 DEBUG Environment variables: None
2016-12-08 00:42:57,500 a6869ecd MainThread object_storage_service(1/1) vm_util.py:287 INFO Running: gcloud compute firewall-rules delete perfkit-firewall-a6869ecd-22-22 --format json --quiet --project REDACTED
2016-12-08 00:43:08,886 a6869ecd MainThread object_storage_service(1/1) vm_util.py:318 DEBUG Ran gcloud compute firewall-rules delete perfkit-firewall-a6869ecd-22-22 --format json --quiet --project REDACTED. Got return code (0).
STDOUT: []

STDERR: Deleted [https://www.googleapis.com/compute/v1/projects/REDACTED/global/firewalls/perfkit-firewall-a6869ecd-22-22].

2016-12-08 00:43:08,886 a6869ecd MainThread object_storage_service(1/1) vm_util.py:284 DEBUG Environment variables: None
2016-12-08 00:43:08,887 a6869ecd MainThread object_storage_service(1/1) vm_util.py:287 INFO Running: gcloud compute firewall-rules describe perfkit-firewall-a6869ecd-22-22 --format json --quiet --project REDACTED
2016-12-08 00:43:09,464 a6869ecd MainThread object_storage_service(1/1) vm_util.py:318 DEBUG Ran gcloud compute firewall-rules describe perfkit-firewall-a6869ecd-22-22 --format json --quiet --project REDACTED. Got return code (1).
STDOUT:
STDERR: ERROR: (gcloud.compute.firewall-rules.describe) Could not fetch resource:
- The resource 'projects/REDACTED/global/firewalls/perfkit-firewall-a6869ecd-22-22' was not found

2016-12-08 00:43:09,464 a6869ecd MainThread object_storage_service(1/1) vm_util.py:284 DEBUG Environment variables: None
2016-12-08 00:43:09,465 a6869ecd MainThread object_storage_service(1/1) vm_util.py:287 INFO Running: gcloud compute firewall-rules delete default-internal-a6869ecd --format json --quiet --project REDACTED
2016-12-08 00:43:25,992 a6869ecd MainThread object_storage_service(1/1) vm_util.py:318 DEBUG Ran gcloud compute firewall-rules delete default-internal-a6869ecd --format json --quiet --project REDACTED. Got return code (0).
STDOUT: []

STDERR: Deleted [https://www.googleapis.com/compute/v1/projects/REDACTED/global/firewalls/default-internal-a6869ecd].

2016-12-08 00:43:25,993 a6869ecd MainThread object_storage_service(1/1) vm_util.py:284 DEBUG Environment variables: None
2016-12-08 00:43:25,994 a6869ecd MainThread object_storage_service(1/1) vm_util.py:287 INFO Running: gcloud compute firewall-rules describe default-internal-a6869ecd --format json --quiet --project REDACTED
2016-12-08 00:43:26,594 a6869ecd MainThread object_storage_service(1/1) vm_util.py:318 DEBUG Ran gcloud compute firewall-rules describe default-internal-a6869ecd --format json --quiet --project REDACTED. Got return code (1).
STDOUT:
STDERR: ERROR: (gcloud.compute.firewall-rules.describe) Could not fetch resource:
- The resource 'projects/REDACTED/global/firewalls/default-internal-a6869ecd' was not found

2016-12-08 00:43:26,595 a6869ecd MainThread object_storage_service(1/1) vm_util.py:284 DEBUG Environment variables: None
2016-12-08 00:43:26,595 a6869ecd MainThread object_storage_service(1/1) vm_util.py:287 INFO Running: gcloud compute networks delete pkb-network-a6869ecd --format json --quiet --project REDACTED
2016-12-08 00:44:13,927 a6869ecd MainThread object_storage_service(1/1) vm_util.py:318 DEBUG Ran gcloud compute networks delete pkb-network-a6869ecd --format json --quiet --project REDACTED. Got return code (0).
STDOUT: []

STDERR: Deleted [https://www.googleapis.com/compute/v1/projects/REDACTED/global/networks/pkb-network-a6869ecd].

2016-12-08 00:44:13,928 a6869ecd MainThread object_storage_service(1/1) vm_util.py:284 DEBUG Environment variables: None
2016-12-08 00:44:13,928 a6869ecd MainThread object_storage_service(1/1) vm_util.py:287 INFO Running: gcloud compute networks describe pkb-network-a6869ecd --format json --quiet --project REDACTED
2016-12-08 00:44:14,522 a6869ecd MainThread object_storage_service(1/1) vm_util.py:318 DEBUG Ran gcloud compute networks describe pkb-network-a6869ecd --format json --quiet --project REDACTED. Got return code (1).
STDOUT:
STDERR: ERROR: (gcloud.compute.networks.describe) Could not fetch resource:
- The resource 'projects/REDACTED/global/networks/pkb-network-a6869ecd' was not found

2016-12-08 00:44:14,526 a6869ecd MainThread object_storage_service(1/1) pkb.py:551 ERROR Benchmark 1/1 object_storage_service (UID: object_storage_service0) failed. Execution will continue.
2016-12-08 00:44:14,540 a6869ecd MainThread pkb.py:642 INFO Benchmark run statuses:
-------------------------------------------------------
Name UID Status
-------------------------------------------------------
object_storage_service object_storage_service0 FAILED
-------------------------------------------------------
Success rate: 0.00% (0/1)
2016-12-08 00:44:14,542 a6869ecd MainThread pkb.py:645 INFO Complete logs can be found at: /tmp/perfkitbenchmarker/runs/a6869ecd/pkb.log
```

I am currently testing the following to see if it works on AWS and confirm if this is only happening on GCP. I will update this issue with my findings after it finishes running.

```
screen ./pkb.py \
--cloud=AWS \
--storage=AWS \
--zones=us-west-2 \
--machine_type=c3.xlarge \
--benchmarks=object_storage_service
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.