mlcommons / mlcommons/modelbench
Cheval running in GCP doesn't spin down idle boxes
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 134
- Forks
- 36
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 17
Description
The deployment doesn't set up the necessary ssh permissions. My first thought is that the annotator VMs should include a public key for which the cheval proxy VM has the private key. Both of which should probably live in the secrets cache.
Also, maybe it's worth looking at whether we could more narrowly scope cheval's secret reading powers to just cheval's secrets. Maybe it needs its own profile?
ERROR:cheval.server:Failed wrangling servers
Traceback (most recent call last):
File "/home/cheval/venv/lib/python3.12/site-packages/cheval/server.py", line 115, in wrangle_servers
self.instance_wrangler.update_servers()
File "/home/cheval/venv/lib/python3.12/site-packages/cheval/server.py", line 33, in wrapper
return f(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/cheval/venv/lib/python3.12/site-packages/cheval/server.py", line 175, in update_servers
changes = self.proposed_changes(containers)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/cheval/venv/lib/python3.12/site-packages/cheval/server.py", line 164, in proposed_changes
if instance.running and instance.idle and self.been_running_a_while:
^^^^^^^^^^^^^
File "/home/cheval/venv/lib/python3.12/site-packages/cheval/core.py", line 258, in idle
activity = self.activity()
^^^^^^^^^^^^^^^
File "/home/cheval/venv/lib/python3.12/site-packages/cheval/core.py", line 297, in activity
text: bytes = self.ssh(*command, capture=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/cheval/venv/lib/python3.12/site-packages/cheval/core.py", line 271, in ssh
raise ValueError(
ValueError: Missing key at /root/.ssh/google_compute_engine. Must run `gcloud compute config-ssh` first
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the deployment configuration and the SSH calls in cheval/server.py and cheval/core.py, then reproduce the gcloud compute config-ssh failure shown in the traceback. Verify how annotator VMs and the proxy obtain keys from the secrets cache, and check whether access can be limited to Cheval's secrets. Done means idle-server wrangling can connect to annotator VMs without the missing-key error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- gcp, python
- Domain
- cloud, infrastructure, security
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100