kestra-io / kestra-io/plugin-gcp
'CPUS' quota issue - can't submit pyspark job with dataproc cluster
- Dominant language
- Java
- Stars
- 15
- Forks
- 27
- Avg merge
- 1d 48m
- Merged PRs (30d)
- 11
Description
### Describe the issue
I am trying to create dataproc cluster to submit pyspark job.
my flow:
```
id: create_dataproc_cluster
namespace: de-project
tasks:
- id: cluster_create
type: io.kestra.plugin.gcp.dataproc.clusters.Create
clusterName: "de-project-cluster"
region: "europe-west1"
zone: "europe-west1-c"
masterMachineType: n1-standard-2
masterDiskSizeGB: 30
workerMachineType: n1-standard-2
workerDiskSizeGB: 30
workers: 2
imageVersion: "2.0-debian10"
bucket: "{{kv('GCP_BUCKET_NAME')}}"
pluginDefaults:
- type: io.kestra.plugin.gcp
values:
projectId: "{{kv('GCP_PROJECT_ID')}}"
location: "{{kv('GCP_LOCATION')}}"
bucket: "{{kv('GCP_BUCKET_NAME')}}"
dataset: "{{kv('GCP_DATASET')}}"
```
got error like this:
```
Operation with name "projects/airbnb-prices-eu/regions/europe-west4/operations/3e33fbe2-bf6f-4cd8-ad08-02a3d7dbf51c" failed with status = GrpcStatusCode{transportCode=UNAVAILABLE} and message = Error Code: UNAVAILABLE, errorSource: COMPUTE_ENGINE, Error Message: The zone 'projects/airbnb-prices-eu/zones/europe-west4-b' does not have enough resources available to fulfill the request. Try a different zone, or try again later.
```
chose about 15 different regions and zones, 90% times faced the same error.
several times I was able to create cluster. but then, when i submit job to that cluster, always receive error about CPU quotas.
I can't increase quota on gcp free trial.
error about CPU:
```
com.google.api.gax.rpc.InvalidArgumentException: Operation with name "projects/airbnb-prices-eu/regions/europe-west4/operations/140cc501-f9d4-44c6-9c5a-c8148c911ca5" failed with status = GrpcStatusCode{transportCode=INVALID_ARGUMENT} and message = Multiple validation errors:
- Insufficient 'CPUS' quota. Requested 12.0, available 7.0. Your resource request exceeds your available quota. See https://cloud.google.com/compute/resource-usage. Use https://cloud.google.com/docs/quotas/view-manage#requesting_higher_quota to request additional quota.
- Insufficient 'CPUS_ALL_REGIONS' quota. Requested 12.0, available 7.0. Your resource request exceeds your available quota. See https://cloud.google.com/compute/resource-usage. Use https://cloud.google.com/docs/quotas/view-manage#requesting_higher_quota to request additional quota.
- This request exceeds CPU quota. Some things to try: request fewer workers (a minimum of 2 is required), use smaller master and/or worker machine types (such as n1-standard-2).
```
my flow for submitting job:
```
id: upload_data_to_bq
namespace: de-project
inputs:
- id: country
type: SELECT
displayName: Select country
values: [spain, portugal, italy, greece, france, germany]
defaults: spain
allowCustomValue: true
tasks:
- id: submit_job
type: io.kestra.plugin.gcp.dataproc.batches.PySparkSubmit
peripherals:
sparkHistoryServer:
dataprocCluster: "de-project-cluster"
runtime:
properties:
spark.driver.cores: "4"
spark.executor.cores: "4"
mainPythonFileUri: "gs://{{ kv('GCP_BUCKET_NAME') }}/code/spark_bq_temp.py"
name: "upload_temp_tables"
region: "europe-west1"
jarFileUris:
- "gs://spark-lib/bigquery/spark-bigquery-latest_2.12.jar"
args:
- country="{{inputs.country}}"
- input_base_path="gs://{{ kv('GCP_BUCKET_NAME') }}"
- output_base_path="{{ kv('GCP_DATASET') }}"
pluginDefaults:
- type: io.kestra.plugin.gcp
values:
projectId: "{{kv('GCP_PROJECT_ID')}}"
location: "{{kv('GCP_LOCATION')}}"
bucket: "{{kv('GCP_BUCKET_NAME')}}"
dataset: "{{kv('GCP_DATASET')}}"
```
when I create dataproc cluster in bash and submit job in bash, works fine.
### Environment
- Kestra Version: develop
Contributor guide
No contributing guide indexed for this repository
Research direction
No repository file or test is named. Start by comparing the Kestra PySparkSubmit flow with the working bash commands and trace the Dataproc cluster and job CPU requests; done means identifying the plugin or configuration difference that causes the quota failure and documenting a reproducible resolution.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- gcp, python, spark
- Domain
- cloud, data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100