kestra-io / kestra-io/plugin-gcp

'CPUS' quota issue - can't submit pyspark job with dataproc cluster

Open
#698 0 comments 1 reaction 0 assignees View on GitHub
area/plugin good first issue
Dominant language
Java
Stars
15
Forks
27
Avg merge
1d 48m
Merged PRs (30d)
11

Description

### Describe the issue

I am trying to create dataproc cluster to submit pyspark job.
my flow:
```
id: create_dataproc_cluster
namespace: de-project

tasks:
- id: cluster_create
type: io.kestra.plugin.gcp.dataproc.clusters.Create
clusterName: "de-project-cluster"
region: "europe-west1"
zone: "europe-west1-c"
masterMachineType: n1-standard-2
masterDiskSizeGB: 30
workerMachineType: n1-standard-2
workerDiskSizeGB: 30
workers: 2
imageVersion: "2.0-debian10"
bucket: "{{kv('GCP_BUCKET_NAME')}}"

pluginDefaults:
- type: io.kestra.plugin.gcp
values:
projectId: "{{kv('GCP_PROJECT_ID')}}"
location: "{{kv('GCP_LOCATION')}}"
bucket: "{{kv('GCP_BUCKET_NAME')}}"
dataset: "{{kv('GCP_DATASET')}}"
```
got error like this:
```
Operation with name "projects/airbnb-prices-eu/regions/europe-west4/operations/3e33fbe2-bf6f-4cd8-ad08-02a3d7dbf51c" failed with status = GrpcStatusCode{transportCode=UNAVAILABLE} and message = Error Code: UNAVAILABLE, errorSource: COMPUTE_ENGINE, Error Message: The zone 'projects/airbnb-prices-eu/zones/europe-west4-b' does not have enough resources available to fulfill the request. Try a different zone, or try again later.
```
chose about 15 different regions and zones, 90% times faced the same error.

several times I was able to create cluster. but then, when i submit job to that cluster, always receive error about CPU quotas.
I can't increase quota on gcp free trial.
error about CPU:
```
com.google.api.gax.rpc.InvalidArgumentException: Operation with name "projects/airbnb-prices-eu/regions/europe-west4/operations/140cc501-f9d4-44c6-9c5a-c8148c911ca5" failed with status = GrpcStatusCode{transportCode=INVALID_ARGUMENT} and message = Multiple validation errors:
- Insufficient 'CPUS' quota. Requested 12.0, available 7.0. Your resource request exceeds your available quota. See https://cloud.google.com/compute/resource-usage. Use https://cloud.google.com/docs/quotas/view-manage#requesting_higher_quota to request additional quota.
- Insufficient 'CPUS_ALL_REGIONS' quota. Requested 12.0, available 7.0. Your resource request exceeds your available quota. See https://cloud.google.com/compute/resource-usage. Use https://cloud.google.com/docs/quotas/view-manage#requesting_higher_quota to request additional quota.
- This request exceeds CPU quota. Some things to try: request fewer workers (a minimum of 2 is required), use smaller master and/or worker machine types (such as n1-standard-2).
```
my flow for submitting job:
```
id: upload_data_to_bq
namespace: de-project

inputs:
- id: country
type: SELECT
displayName: Select country
values: [spain, portugal, italy, greece, france, germany]
defaults: spain
allowCustomValue: true

tasks:
- id: submit_job
type: io.kestra.plugin.gcp.dataproc.batches.PySparkSubmit
peripherals:
sparkHistoryServer:
dataprocCluster: "de-project-cluster"
runtime:
properties:
spark.driver.cores: "4"
spark.executor.cores: "4"
mainPythonFileUri: "gs://{{ kv('GCP_BUCKET_NAME') }}/code/spark_bq_temp.py"
name: "upload_temp_tables"
region: "europe-west1"
jarFileUris:
- "gs://spark-lib/bigquery/spark-bigquery-latest_2.12.jar"
args:
- country="{{inputs.country}}"
- input_base_path="gs://{{ kv('GCP_BUCKET_NAME') }}"
- output_base_path="{{ kv('GCP_DATASET') }}"

pluginDefaults:
- type: io.kestra.plugin.gcp
values:
projectId: "{{kv('GCP_PROJECT_ID')}}"
location: "{{kv('GCP_LOCATION')}}"
bucket: "{{kv('GCP_BUCKET_NAME')}}"
dataset: "{{kv('GCP_DATASET')}}"
```

when I create dataproc cluster in bash and submit job in bash, works fine.

### Environment

- Kestra Version: develop

Contributor guide

No contributing guide indexed for this repository

Research direction

No repository file or test is named. Start by comparing the Kestra PySparkSubmit flow with the working bash commands and trace the Dataproc cluster and job CPU requests; done means identifying the plugin or configuration difference that causes the quota failure and documenting a reproducible resolution.

Written by the indexing model from the issue text.

Assessment

Tech stack
gcp, python, spark
Domain
cloud, data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.