mlcommons / mlcommons/modelbench

Cheval running in GCP doesn't spin down idle boxes

Open
#953 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
134
Forks
36
Avg merge
1d 11h
Merged PRs (30d)
17

Description

The deployment doesn't set up the necessary ssh permissions. My first thought is that the annotator VMs should include a public key for which the cheval proxy VM has the private key. Both of which should probably live in the secrets cache.

Also, maybe it's worth looking at whether we could more narrowly scope cheval's secret reading powers to just cheval's secrets. Maybe it needs its own profile?

ERROR:cheval.server:Failed wrangling servers
Traceback (most recent call last):
  File "/home/cheval/venv/lib/python3.12/site-packages/cheval/server.py", line 115, in wrangle_servers
    self.instance_wrangler.update_servers()
  File "/home/cheval/venv/lib/python3.12/site-packages/cheval/server.py", line 33, in wrapper
    return f(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cheval/venv/lib/python3.12/site-packages/cheval/server.py", line 175, in update_servers
    changes = self.proposed_changes(containers)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cheval/venv/lib/python3.12/site-packages/cheval/server.py", line 164, in proposed_changes
    if instance.running and instance.idle and self.been_running_a_while:
                            ^^^^^^^^^^^^^
  File "/home/cheval/venv/lib/python3.12/site-packages/cheval/core.py", line 258, in idle
    activity = self.activity()
               ^^^^^^^^^^^^^^^
  File "/home/cheval/venv/lib/python3.12/site-packages/cheval/core.py", line 297, in activity
    text: bytes = self.ssh(*command, capture=True)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cheval/venv/lib/python3.12/site-packages/cheval/core.py", line 271, in ssh
    raise ValueError(
ValueError: Missing key at /root/.ssh/google_compute_engine. Must run `gcloud compute config-ssh` first

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the deployment configuration and the SSH calls in cheval/server.py and cheval/core.py, then reproduce the gcloud compute config-ssh failure shown in the traceback. Verify how annotator VMs and the proxy obtain keys from the secrets cache, and check whether access can be limited to Cheval's secrets. Done means idle-server wrangling can connect to annotator VMs without the missing-key error.

Written by the indexing model from the issue text.

Assessment

Tech stack
gcp, python
Domain
cloud, infrastructure, security
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.