kestra-io / kestra-io/plugin-gcp

Dataflow tasks never infer projectId from the service-account key

Open
#697 0 comments 0 reactions 0 assignees View on GitHub
area/plugin good first issue
Dominant language
Java
Stars
15
Forks
27
Avg merge
1d 48m
Merged PRs (30d)
11

Description

### What

Every task in `io.kestra.plugin.gcp.dataflow` reads `projectId` with `orElseThrow()` **before** building the Dataflow client, and `AbstractDataflow.dataflowClient(...)` resolves credentials through the static `CredentialService.connection(...)` path, which does not write the inferred project id back onto the task. As a result a Dataflow task requires an explicit `projectId` even when the configured `serviceAccount` key carries one — unlike every other GCP task, which infers it.

Affected: `CancelJob`, `GetJob`, `WaitForJob`, `LaunchTemplate`, `LaunchFlexTemplate`, and `Trigger` (each reads `runContext.render(this.projectId)...orElseThrow()` before `dataflowClient(runContext)`).

### Why this is a separate change

Surfaced during the plugin-gcp-lib shared-kernel review (#696). It was intentionally **not** bundled into that extraction PR because making `projectId` inferable is a user-facing contract change (the property goes from effectively required to optional for Dataflow), which deserves to be a declared change rather than slipped into a mechanical repoint. It also requires reworking the current hermetic `DataflowTest` (which mocks `dataflowClient`).

### Proposed fix

- Route the instance `AbstractDataflow.dataflowClient(runContext)` through `this.credentials(runContext)` so the resolved project id is written back (matching the other GCP `AbstractTask` subclasses).
- In each Dataflow task/trigger, resolve credentials (build the client) before reading `projectId`, keeping `orElseThrow()` so a genuinely-unresolvable project id still fails clearly.
- Update `DataflowTest` accordingly.

### Acceptance

A Dataflow task with a `serviceAccount` key that carries a project id, and no explicit `projectId`, runs against that inferred project instead of failing with `NoSuchElementException`.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with AbstractDataflow.dataflowClient(runContext), the listed Dataflow task and trigger classes, and DataflowTest. Trace how credentials resolve and how projectId is read before and after client creation, then update the tests to cover a service-account key carrying a project id. Done means a task without explicit projectId uses the inferred project and still fails clearly when no project id can be resolved.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, java
Domain
backend, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.