apache / apache/airflow

cluster config generation using cluster config YAML file.

Open
#17,089 8 comments 0 reactions 0 assignees View on GitHub
good first issue kind:feature provider:google
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

**Description**

Add cluster config generation from yaml file which is generated using `gcloud dataproc clusters export` command.

**Use case / motivation**

We are generating quite a lot of clusters using a cluster config which keeps on changing depending upon project to project. I wrote custom operator which reads yaml config and converts it to dict and then uses the same config to genreate cluster. Wondering if it will be useful if cluster config generator operator can accept a yaml config or can create a new operator.

**Are you willing to submit a PR?**

I do have a working code for converting yaml stored on gcs to dict.

**Related Issues**

There is no issue as such, it is a functionality which allows users to specify cluster config as yaml file.

Contributor guide

Open the contributing guide

Research direction

Start by locating Airflow's existing cluster config generator and comparing its inputs with the YAML produced by `gcloud dataproc clusters export`. Define whether the generator should accept a YAML file from GCS or introduce a new operator, then verify that the resulting configuration creates the same cluster settings as the exported YAML.

Written by the indexing model from the issue text.

Assessment

Tech stack
gcp, python
Domain
cloud, data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.