cloud-bulldozer / cloud-bulldozer/perfscale-managed-services

Controlling installation result of osde2e with exit code is not enough

Open
#73 0 comments 0 reactions 0 assignees View on GitHub
bug osde2e P2
Dominant language
Python
Stars
2
Forks
11
PR merge metrics
No merged PRs in 30d

Description

Installing with a wrong ocm token results on thread exiting with exit code 0

```
2021-03-10 10:51:26.081 INFO osde2e-wrapper - main: Attempting to start 10 clusters with 2 batch size
2021-03-10 10:51:26.089 INFO osde2e-wrapper - _build_cluster: Started cluster osde2e-o3z-0001 (1 of 10)
2021-03-10 10:51:26.092 INFO osde2e-wrapper - _build_cluster: Started cluster osde2e-o3z-0002 (2 of 10)
2021-03-10 10:51:26.544 INFO osde2e-wrapper - _watcher: Requested Clusters for test 22b46690-9244-4a5d-935e-3c48f27a6338: 0
2021-03-10 10:51:31.888 INFO osde2e-wrapper - _build_cluster: Attempting to load metadata json
2021-03-10 10:51:31.888 ERROR osde2e-wrapper - _build_cluster: [Errno 2] No such file or directory: '/tmp/22b46690-9244-4a5d-935e-3c48f27a6338/osde2e-o3z-0002/metadata.json'
2021-03-10 10:51:31.889 ERROR osde2e-wrapper - _build_cluster: Failed to load metadata.json file located /tmp/22b46690-9244-4a5d-935e-3c48f27a6338/osde2e-o3z-0002
2021-03-10 10:51:31.889 INFO osde2e-wrapper - _download_kubeconfig: Attempting to load metadata json
2021-03-10 10:51:31.890 ERROR osde2e-wrapper - _download_kubeconfig: Failed to load cluster-id from metadata.json file located on /tmp/22b46690-9244-4a5d-935e-3c48f27a6338/osde2e-o3z-0002, kubeconfig file wont be downloaded
```

Installation logs:

```
2021/03/10 10:51:26 addon timeout is 30
2021/03/10 10:51:26 load.go:57: Custom YAML config provided, loading from cluster_account.yaml
2021/03/10 10:51:26 e2e.go:260: Could not create reporter directory: mkdir /tmp/22b46690-9244-4a5d-935e-3c48f27a6338/osde2e-o3z-0001: file exists
2021/03/10 10:51:26 e2e.go:303: Outputting log to build log at /tmp/22b46690-9244-4a5d-935e-3c48f27a6338/osde2e-o3z-0001/test_output.log
2021/03/10 10:51:26 versions.go:34: Querying cluster versions endpoint.
2021/03/10 10:51:26 go_logger.go:152: OCM auth: failed to get tokens, got http code 400, will not attempt to retry. err: invalid_grant: Offline user session not found
2021/03/10 10:51:26 retryer.go:23: error during OCM attempt: can't get access token: invalid_grant: Offline user session not found
2021/03/10 10:51:30 go_logger.go:152: OCM auth: failed to get tokens, got http code 400, will not attempt to retry. err: invalid_grant: Offline user session not found
2021/03/10 10:51:30 retryer.go:23: error during OCM attempt: can't get access token: invalid_grant: Offline user session not found
2021/03/10 10:51:31 go_logger.go:152: OCM auth: failed to get tokens, got http code 400, will not attempt to retry. err: invalid_grant: Offline user session not found
2021/03/10 10:51:31 retryer.go:23: error during OCM attempt: can't get access token: invalid_grant: Offline user session not found
```

[This conditional](https://github.com/cloud-bulldozer/perfscale-managed-services/blob/803e35a0f196d317b25e6d2288e1fbb5980d6edf/osde2e/osde2e-wrapper.py#L131) is not being executed, as we dont see the "Failed to build cluster" logging.

We need to fix the failed cluster installation detection but also assigning a default cluster-id as "not generated" or something if we detect that metadata.json file is not generated or generated without cluster-id

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in osde2e/osde2e-wrapper.py at the linked conditional around line 131, then trace how installation results and metadata.json are handled. Reproduce the wrong-token case using the logged failure, and verify that failed cluster installation is detected and logged while a missing or incomplete metadata.json receives a default cluster-id such as "not generated".

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.