apache / apache/hop

[Feature Request]: Add gcpTempLocation option to Direct & Dataflow Runner configuration

Open
#2,355 0 comments 0 reactions 0 assignees View on GitHub
awaiting triage Beam P2
Dominant language
Java
Stars
1.5k
Forks
476
Avg merge
19h
Merged PRs (30d)
227

Description

### What would you like to happen?

For some transforms such as a bigquery load the Temp location has to be a GCS path.
Beam has a differentiation between "gcpTempLocation" and "tempLocation" for IO's that require the data to be in gcs

### Issue Priority

Priority: 2

### Issue Component

Component: Beam

Contributor guide

Open the contributing guide

Research direction

Start by locating the Direct and Dataflow Runner configuration and the existing tempLocation handling. Compare Beam's gcpTempLocation and tempLocation distinction for transforms such as BigQuery loads, then verify that the new option reaches the runner configuration and supports a GCS path.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, java
Domain
cloud, data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.