kestra-io / kestra-io/plugin-gcp
Dataflow tasks never infer projectId from the service-account key
- Dominant language
- Java
- Stars
- 15
- Forks
- 27
- Avg merge
- 1d 48m
- Merged PRs (30d)
- 11
Description
### What
Every task in `io.kestra.plugin.gcp.dataflow` reads `projectId` with `orElseThrow()` **before** building the Dataflow client, and `AbstractDataflow.dataflowClient(...)` resolves credentials through the static `CredentialService.connection(...)` path, which does not write the inferred project id back onto the task. As a result a Dataflow task requires an explicit `projectId` even when the configured `serviceAccount` key carries one — unlike every other GCP task, which infers it.
Affected: `CancelJob`, `GetJob`, `WaitForJob`, `LaunchTemplate`, `LaunchFlexTemplate`, and `Trigger` (each reads `runContext.render(this.projectId)...orElseThrow()` before `dataflowClient(runContext)`).
### Why this is a separate change
Surfaced during the plugin-gcp-lib shared-kernel review (#696). It was intentionally **not** bundled into that extraction PR because making `projectId` inferable is a user-facing contract change (the property goes from effectively required to optional for Dataflow), which deserves to be a declared change rather than slipped into a mechanical repoint. It also requires reworking the current hermetic `DataflowTest` (which mocks `dataflowClient`).
### Proposed fix
- Route the instance `AbstractDataflow.dataflowClient(runContext)` through `this.credentials(runContext)` so the resolved project id is written back (matching the other GCP `AbstractTask` subclasses).
- In each Dataflow task/trigger, resolve credentials (build the client) before reading `projectId`, keeping `orElseThrow()` so a genuinely-unresolvable project id still fails clearly.
- Update `DataflowTest` accordingly.
### Acceptance
A Dataflow task with a `serviceAccount` key that carries a project id, and no explicit `projectId`, runs against that inferred project instead of failing with `NoSuchElementException`.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with AbstractDataflow.dataflowClient(runContext), the listed Dataflow task and trigger classes, and DataflowTest. Trace how credentials resolve and how projectId is read before and after client creation, then update the tests to cover a service-account key carrying a project id. Done means a task without explicit projectId uses the inferred project and still fails clearly when no project id can be resolved.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud, java
- Domain
- backend, cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100