hashicorp / hashicorp/vagrant_cloud

`vagrant cloud search` is slow due to `O(n+1)` API requests

Open
#80 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Ruby
Stars
16
Forks
13
PR merge metrics
No merged PRs in 30d

Description

A simple call to `vagrant cloud search ubuntu2204` takes upwards of 15 seconds to run on high end hardware... adding `--debug-timestamp` I noticed it hangs on API requests and when I searched the debug output I noticed the following for one search:

```log
2022-10-26 00:39:36 [ INFO] excon REQUEST method=:get identifier="fb82e6ec-bf46-44de-b432-369cc1a6e8cd" url="https://vagrantcloud.com/api/v1/search" query="q: ubuntu2204" headers="Host: vagrantcloud.com:443, Accept: application/json, Content-Type: application/json, X-Request-Id: fb82e6ec-bf46-44de-b432-369cc1a6e8cd" duration="83ms"
2022-10-26 00:39:36 [ INFO] excon REQUEST method=:get identifier="9b6f83e7-f51c-45a8-b4b3-e0a23250937a" url="https://vagrantcloud.com/api/v1/user/generic" headers="Host: vagrantcloud.com:443, Accept: application/json, Content-Type: application/json, X-Request-Id: 9b6f83e7-f51c-45a8-b4b3-e0a23250937a" duration="40ms"
2022-10-26 00:39:40 [ INFO] excon REQUEST method=:get identifier="48e70e0b-83f1-4e9e-b445-ba9d786c43d1" url="https://vagrantcloud.com/api/v1/user/roboxes" headers="Host: vagrantcloud.com:443, Accept: application/json, Content-Type: application/json, X-Request-Id: 48e70e0b-83f1-4e9e-b445-ba9d786c43d1" duration="34ms"
2022-10-26 00:39:44 [ INFO] excon REQUEST method=:get identifier="7728c2dc-9f5f-4df0-8dcf-6e95a02aeaa2" url="https://vagrantcloud.com/api/v1/user/fasmat" headers="Host: vagrantcloud.com:443, Accept: application/json, Content-Type: application/json, X-Request-Id: 7728c2dc-9f5f-4df0-8dcf-6e95a02aeaa2" duration="34ms"
2022-10-26 00:39:44 [ INFO] excon REQUEST method=:get identifier="212100db-e3b4-4240-96c4-19c556e59cca" url="https://vagrantcloud.com/api/v1/user/techchad2022" headers="Host: vagrantcloud.com:443, Accept: application/json, Content-Type: application/json, X-Request-Id: 212100db-e3b4-4240-96c4-19c556e59cca" duration="39ms"
2022-10-26 00:39:45 [ INFO] excon REQUEST method=:get identifier="b87e3ea7-5134-4b9b-8f50-e7b502522a9a" url="https://vagrantcloud.com/api/v1/user/ilionx" headers="Host: vagrantcloud.com:443, Accept: application/json, Content-Type: application/json, X-Request-Id: b87e3ea7-5134-4b9b-8f50-e7b502522a9a" duration="41ms"
2022-10-26 00:39:46 [ INFO] excon REQUEST method=:get identifier="61896816-1168-4462-b217-9a1b3e2d3c69" url="https://vagrantcloud.com/api/v1/user/jaredeh" headers="Host: vagrantcloud.com:443, Accept: application/json, Content-Type: application/json, X-Request-Id: 61896816-1168-4462-b217-9a1b3e2d3c69" duration="42ms"
2022-10-26 00:39:46 [ INFO] excon REQUEST method=:get identifier="34620a6d-acb7-4ae1-8184-0a88d79b2db4" url="https://vagrantcloud.com/api/v1/user/krec" headers="Host: vagrantcloud.com:443, Accept: application/json, Content-Type: application/json, X-Request-Id: 34620a6d-acb7-4ae1-8184-0a88d79b2db4" duration="43ms"
2022-10-26 00:39:47 [ INFO] excon REQUEST method=:get identifier="540a21a4-7ae4-48fd-8749-8a24c742dd0b" url="https://vagrantcloud.com/api/v1/user/marcinbojko" headers="Host: vagrantcloud.com:443, Accept: application/json, Content-Type: application/json, X-Request-Id: 540a21a4-7ae4-48fd-8749-8a24c742dd0b" duration="41ms"
2022-10-26 00:39:47 [ INFO] excon REQUEST method=:get identifier="5df81256-6c75-4bb2-be84-135886224fea" url="https://vagrantcloud.com/api/v1/user/maroskukan" headers="Host: vagrantcloud.com:443, Accept: application/json, Content-Type: application/json, X-Request-Id: 5df81256-6c75-4bb2-be84-135886224fea" duration="38ms"
```
Search results:
```log
2022-10-26 00:39:48 [ INFO] info: | NAME | VERSION | DOWNLOADS | PROVIDERS |
2022-10-26 00:39:48 [ INFO] info: +----------------------------+----------------+-----------+--------------------------------------------------------+
2022-10-26 00:39:48 [ INFO] info: | generic/ubuntu2204 | 4.1.18 | 26,057 | vmware_desktop, virtualbox, parallels, libvirt, hyperv |
2022-10-26 00:39:48 [ INFO] info: | roboxes/ubuntu2204 | 4.1.18 | 1,891 | virtualbox, parallels, vmware_desktop, hyperv, libvirt |
2022-10-26 00:39:48 [ INFO] info: | fasmat/ubuntu2204-desktop | 22.0509.1 | 334 | virtualbox, vmware_desktop |
2022-10-26 00:39:48 [ INFO] info: | techchad2022/ubuntu2204 | 0.2 | 187 | virtualbox, libvirt |
2022-10-26 00:39:48 [ INFO] info: | ilionx/ubuntu2204-minikube | 1.1.0-20220825 | 84 | virtualbox |
2022-10-26 00:39:48 [ INFO] info: | techchad2022/lamp | 0.2 | 74 | virtualbox, libvirt |
2022-10-26 00:39:48 [ INFO] info: | jaredeh/ubuntu2204-server | 0.1.0 | 71 | libvirt |
2022-10-26 00:39:48 [ INFO] info: | krec/ubuntu2204-german | 20220904.0.1 | 71 | virtualbox |
2022-10-26 00:39:48 [ INFO] info: | marcinbojko/ubuntu2204 | 1.0.0 | 53 | virtualbox |
2022-10-26 00:39:48 [ INFO] info: | maroskukan/ubuntu2204 | 2022.07.26 | 36 | virtualbox, hyperv |
2022-10-26 00:39:48 [ INFO] info: +----------------------------+----------------+-----------+--------------------------------------------------------+
```

It appears that the search is performing a lookup on each user/org in the result set - thus `O(n+1)` complexity. I checked the timestamps and 12 seconds elapses largely due to connection overhead.

I don't see anything in the output that would necessitate the user lookup... I assume that's for some other consumer of these APIs? Am I missing something that requires the extra lookups? Perhaps a workaround for stale data in the initial query response?

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.