google-deepmind / google-deepmind/xmanager

ResourceExhausted: 429 The following quota metrics exceed quota limits

Open
#29 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
921
Forks
69
Avg merge
1d 1h
Merged PRs (30d)
1

Description

Hi! Thanks for building this amazing project. Recently I'm running script on xmanager+vertex.AI on TPU v2 and v3, but I keep getting this error:
```
google.api_core.exceptions.ResourceExhausted: 429 The following quota metrics exceed quota limits: aiplatform.googleapis.com/custom_model_training_tpu_v2
```

The error is thrown at this line - https://github.com/deepmind/xmanager/blob/v0.2.0/xmanager/cloud/vertex.py#L181.

Below are the sanity checks that I've done:
* I found the service account [here](https://github.com/deepmind/xmanager/blob/v0.2.0/xmanager/cloud/vertex.py#L169) can be loaded nicely, tho it would soon be assigned to `None [here](https://github.com/deepmind/xmanager/blob/v0.2.0/xmanager/cloud/vertex.py#L172-L173) as I'm requesting TPU v2 or v3.
* `tensorboard` is set to empty string.
* the `self.location`, `self.project`, `pools` and `auth.get_bucket()` all look good. where the location is `us-central1`, and `pools` showing --
```
[machine_spec {
machine_type: "cloud-tpu"
accelerator_type: TPU_V2
accelerator_count: 8
}
```

I've enabled the three APIs mentioned in the readme (`IAM`, `Cloud AI Platform`, `Container Registry`), additionally `Vertex API` and `Cloud Resource Manager API` was enabled. I also checked the Quota page on the console, which looks fine as well. Doesn't look like I'm overusing the resources as described in the error message "exceed quota limits".

It's been bugging me for quite a few days, and would be really appreciated if anyone could suggest what's possibly going on there. Thanks in advance!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.