iterative / iterative/cml.dev

Do we need different credential for creating runner and access GCS

Open
#386 3 comments 1 reaction 0 assignees View on GitHub
documentation
Dominant language
TypeScript
Stars
13
Forks
22
PR merge metrics
No merged PRs in 30d

Description

I have the following GitHub action file,
```sh
jobs:
deploy-runner:
runs-on: ubuntu-20.04
steps:
- uses: iterative/setup-cml@v1
- uses: actions/checkout@v2
- name: Deploy runner on GCP
env:
REPO_TOKEN: ${{ secrets.REPO_ACCESS_TOKEN }}
GOOGLE_APPLICATION_CREDENTIALS_DATA: ${{ secrets.GOOGLE_APPLICATION_CREDENTIALS_DATA }}
run: |
cml runner \
--cloud=gcp \
--cloud-region=us-central1-a \
--cloud-type=n1-highmem-2 \
--labels=gpu_runner \
--single=true \
--cloud-gpu=nvidia-tesla-t4 \
--cloud-spot=false \
--cloud-hdd-size=128

training:
name: Training and Reporting
needs:
- deploy-runner
runs-on: [self-hosted, gpu_runner]
steps:
- uses: actions/checkout@v2
- uses: iterative/setup-cml@v1
- uses: iterative/setup-dvc@v1
- uses: actions/setup-python@v2
with:
python-version: '3.10'
- name: Training
env:
repo_token: ${{ secrets.REPO_ACCESS_TOKEN }}
GOOGLE_APPLICATION_CREDENTIALS_DATA: ${{ secrets.GOOGLE_APPLICATION_CREDENTIALS_DATA }}
run: |
# Pull dataset with DVC
dvc pull data

# Reproduce pipeline if any changes detected in dependencies
dvc repro
```
The `deploy-runner:` job working without any issue but `training` job fail in `dvc pull data` and it's give the following error
```sh
_request out of retries on exception: ('Failed to retrieve http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true from the Google Compute Engine metadata service. Status: 404 Response:\nb\'\\n\\n \\n \\n Error 404 (Not Found)!!1\\n \\n ****margin:0;padding:0***html,code***font:15px/22px arial,sans-serif***html***background:#fff;color:#222;padding:15px***body***margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px**** > body***background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px***p***margin:11px 0 22px;overflow:hidden***ins***color:#777;text-decoration:none***a img***border:0***@media screen and (max-width:772px)***body***background:none;margin-top:0;max-width:none;padding-right:0***#logo***background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px***@media only screen and (min-resolution:192dpi)***#logo***background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0***@media only screen and (-webkit-min-device-pixel-ratio:2)***#logo***background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%***#logo***display:inline-block;height:54px;width:150px***\\n \\n \\n

404. That\\xe2\\x80\\x99s an error.\\n

The requested URL /computeMetadata/v1/instance/service-accounts/default/?recursive=true was not found on this server. That\\xe2\\x80\\x99s all we know.\\n\'', )
Traceback (most recent call last):
File "google/auth/compute_engine/credentials.py", line 111, in refresh
File "google/auth/compute_engine/credentials.py", line 87, in _retrieve_info
File "google/auth/compute_engine/_metadata.py", line 234, in get_service_account_info
File "google/auth/compute_engine/_metadata.py", line 182, in get
google.auth.exceptions.TransportError: ('Failed to retrieve http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true from the Google Compute Engine metadata service. Status: 404 Response:\nb\'\\n\\n \\n \\n Error 404 (Not Found)!!1\\n \\n ****margin:0;padding:0***html,code***font:15px/22px arial,sans-serif***html***background:#fff;color:#222;padding:15px***body***margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px**** > body***background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px***p***margin:11px 0 22px;overflow:hidden***ins***color:#777;text-decoration:none***a img***border:0***@media screen and (max-width:772px)***body***background:none;margin-top:0;max-width:none;padding-right:0***#logo***background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px***@media only screen and (min-resolution:192dpi)***#logo***background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0***@media only screen and (-webkit-min-device-pixel-ratio:2)***#logo***background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%***#logo***display:inline-block;height:54px;width:150px***\\n \\n \\n

404. That\\xe2\\x80\\x99s an error.\\n

The requested URL /computeMetadata/v1/instance/service-accounts/default/?recursive=true was not found on this server. That\\xe2\\x80\\x99s all we know.\\n\'', )

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "gcsfs/retry.py", line 115, in retry_request
File "gcsfs/core.py", line 374, in _request
File "gcsfs/core.py", line 353, in _get_headers
File "gcsfs/credentials.py", line 182, in apply
File "gcsfs/credentials.py", line 177, in maybe_refresh
File "google/auth/compute_engine/credentials.py", line 117, in refresh
File "", line 3, in raise_from
google.auth.exceptions.RefreshError: ('Failed to retrieve http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true from the Google Compute Engine metadata service. Status: 404 Response:\nb\'\\n\\n \\n \\n Error 404 (Not Found)!!1\\n \\n ****margin:0;padding:0***html,code***font:15px/22px arial,sans-serif***html***background:#fff;color:#222;padding:15px***body***margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px**** > body***background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px***p***margin:11px 0 22px;overflow:hidden***ins***color:#777;text-decoration:none***a img***border:0***@media screen and (max-width:772px)***body***background:none;margin-top:0;max-width:none;padding-right:0***#logo***background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px***@media only screen and (min-resolution:192dpi)***#logo***background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0***@media only screen and (-webkit-min-device-pixel-ratio:2)***#logo***background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%***#logo***display:inline-block;height:54px;width:150px***\\n \\n \\n

404. That\\xe2\\x80\\x99s an error.\\n

The requested URL /computeMetadata/v1/instance/service-accounts/default/?recursive=true was not found on this server. That\\xe2\\x80\\x99s all we know.\\n\'', )
ERROR: unexpected error - ('Failed to retrieve http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true from the Google Compute Engine metadata service. Status: 404 Response:\nb\'\\n\\n \\n \\n Error 404 (Not Found)!!1\\n \\n ****margin:0;padding:0***html,code***font:15px/22px arial,sans-serif***html***background:#fff;color:#222;padding:15px***body***margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px**** > body***background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px***p***margin:11px 0 22px;overflow:hidden***ins***color:#777;text-decoration:none***a img***border:0***@media screen and (max-width:772px)***body***background:none;margin-top:0;max-width:none;padding-right:0***#logo***background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px***@media only screen and (min-resolution:192dpi)***#logo***background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0***@media only screen and (-webkit-min-device-pixel-ratio:2)***#logo***background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%***#logo***display:inline-block;height:54px;width:150px***\\n \\n \\n

404. That\\xe2\\x80\\x99s an error.\\n

The requested URL /computeMetadata/v1/instance/service-accounts/default/?recursive=true was not found on this server. That\\xe2\\x80\\x99s all we know.\\n\'', ): ('Failed to retrieve http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true from the Google Compute Engine metadata service. Status: 404 Response:\nb\'\\n\\n \\n \\n Error 404 (Not Found)!!1\\n \\n ****margin:0;padding:0***html,code***font:15px/22px arial,sans-serif***html***background:#fff;color:#222;padding:15px***body***margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px**** > body***background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px***p***margin:11px 0 22px;overflow:hidden***ins***color:#777;text-decoration:none***a img***border:0***@media screen and (max-width:772px)***body***background:none;margin-top:0;max-width:none;padding-right:0***#logo***background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px***@media only screen and (min-resolution:192dpi)***#logo***background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0***@media only screen and (-webkit-min-device-pixel-ratio:2)***#logo***background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%***#logo***display:inline-block;height:54px;width:150px***\\n \\n \\n

404. That\\xe2\\x80\\x99s an error.\\n

The requested URL /computeMetadata/v1/instance/service-accounts/default/?recursive=true was not found on this server. That\\xe2\\x80\\x99s all we know.\\n\'', )

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
```
According to the documentation [cloud-storage-provider-credentials]( https://cml.dev/doc/cml-with-dvc?tab=Google-Cloud-Storage#cloud-storage-provider-credentials) the `GCS` environment variable is `GOOGLE_APPLICATION_CREDENTIALS` but i'm using `GOOGLE_APPLICATION_CREDENTIALS_DATA` is this the issue or i'm missing something

Thanks

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.