GoogleCloudPlatform / GoogleCloudPlatform/python-docs-samples

Composer_dags.py script (pause & unpause dags) is taking hours to finish

Offen
#10,805 3 Kommentare 0 Reaktionen 1 zugewiesene Person Zugewiesen an @rafalbiegacz Auf GitHub ansehen
api: composer priority: p2 samples type: bug
Vorherrschende Sprache
Jupyter Notebook
Sterne
8.1k
Forks
6.7k
Ø Merge
4 T. 2 Std.
Gemergte PRs (30 T.)
16

Beschreibung

## In which file did you encounter the issue?
composer/tools/composer_dags.py

## Did you change the file? If so, how?
No change

## Describe the issue
I run the script for the environment of 500 dags. It is extremely slow, it takes more than 1-2min to do this for each dag. I has taken almost entire day and not yet finished. Is there a better way to batch those requests and pause multiple dags at once?
This has to be done before the snapshot can be created.

Example logs:
2023-10-25 15:49:30,718 - Executing shell command: CLOUDSDK_API_ENDPOINT_OVERRIDES_COMPOSER=https://composer.googleapis.com/ gcloud composer environments run composer-dev --project=whejna-dataops --location=europe-west3 dags pause -- export_task_reschedule_v1_0
Executing the command: [ airflow dags pause export_task_reschedule_v1_0 ]...
Command has been started. execution_id=3fa888b4-7cf4-4c0e-ab07-c54cccda24ac
Use ctrl-c to interrupt the command
2023-10-25 15:51:07,371 - Dag: export_task_reschedule_v1_0, paused: True

Thanks

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.