[helm] Workload v2 Launcher ignores storage.gcs.credentialsJsonPath when building job pods causing NoSuchFileException
- Lingua principale
- Python
- Stelle
- 22.1k
- Fork
- 5.3k
- Metriche di merge delle PR
- Metriche PR in attesa
Descrizione
### Helm Chart Version
Airbyte App Version: 2.0.1 (Helm chart 2.0.18)
### What step the error happened?
Other
### Relevant information
When deploying Airbyte on Kubernetes using the v2 Helm chart and enabling external GCS logging, there is a mismatch between how the Helm chart configures the environment variables and how the Workload API builds the volume mounts for dynamic job pods.
1. Deploy Airbyte with the v2 architecture.
2. Configure external GCS logging/storage in `values.yaml` and set a custom `credentialsJsonPath`:
```yaml
storage:
type: gcs
secretName: airbyte-mounted-secrets
gcs:
projectId: my-gcp-project
credentialsJsonSecretKey: airbyte-gcp-json
credentialsJsonPath: "/secrets/gcs-log-creds/gcp.json"
```
3. Trigger a connection check or sync in the UI.
4. Inspect the generated job pod (e.g., `rce-postgres-check-...`).
**The Bug:**
The Workload Launcher passes the `GOOGLE_APPLICATION_CREDENTIALS` environment variable exactly as defined in `values.yaml` (e.g., `/secrets/gcs-log-creds/gcp.json`). However, it completely ignores this path when generating the Kubernetes manifest's `volumeMounts`, instead hardcoding the secret mount path to `/secrets/gcp-creds/`.
Because the `connector-sidecar` container is looking for a directory that the Workload Launcher didn't create, it immediately crashes.
**Current Workaround:**
Users must remove the `credentialsJsonPath` configuration entirely from their `values.yaml` and rely solely on `credentialsJsonSecretKey`. This forces the system to fall back to the Workload Launcher's hardcoded `/secrets/gcp-creds/` directory, though this contradicts older official documentation. The Helm chart should either respect `credentialsJsonPath` for dynamic pods or deprecate the field.
This may also be related to: https://github.com/airbytehq/airbyte/issues/70352
### Relevant log output
```shell
2026-02-24 00:44:59,031 [main] INFO i.m.r.Micronaut(start):183 - No embedded container found. Running as CLI application
Exception in thread "main" io.micronaut.context.exceptions.BeanInstantiationException: Error instantiating bean of type [io.airbyte.workers.workload.WorkloadOutputWriter]
Message: /secrets/gcs-log-creds/gcp.json
Path Taken:
new @j.i.Singleton i.a.c.ConnectorWatcher(@Named("output") Path outputPath, @Named("configDir") String configDir, @Value int fileTimeoutMinutes, @Value int fileTimeoutMinutesWithinSync, SidecarInput sidecarInput, ConnectorMessageProcessor connectorMessageProcessor, AirbyteMessageSerDeProvider serDeProvider, AirbyteProtocolVersionedMigratorFactory airbyteProtocolVersionedMigratorFactory, GsonPksExtractor gsonPksExtractor, WorkloadApiClient workloadApiClient, WorkloadOutputWriter outputWriter, SidecarLogContextFactory logContextFactory, HeartbeatMonitor heartbeatMonitor, MetricClient metricClient)
\---> new @j.i.Singleton i.a.c.ConnectorWatcher(@Named("output") Path outputPath, @Named("configDir") String configDir, @Value int fileTimeoutMinutes, @Value int fileTimeoutMinutesWithinSync, SidecarInput sidecarInput, ConnectorMessageProcessor connectorMessageProcessor, AirbyteMessageSerDeProvider serDeProvider, AirbyteProtocolVersionedMigratorFactory airbyteProtocolVersionedMigratorFactory, GsonPksExtractor gsonPksExtractor, WorkloadApiClient workloadApiClient, [WorkloadOutputWriter outputWriter], SidecarLogContextFactory logContextFactory, HeartbeatMonitor heartbeatMonitor, MetricClient metricClient)
\---> new @j.i.Singleton i.a.w.w.WorkloadOutputWriter([@Named("outputDocumentStore") StorageClient storageClient], AirbyteApiClient airbyteApiClient, MetricClient metricClient)
... [TRUNCATED STACK TRACE] ...
Caused by: java.nio.file.NoSuchFileException: /secrets/gcs-log-creds/gcp.json
at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:261)
at java.base/java.nio.file.Files.newByteChannel(Files.java:380)
at java.base/java.nio.file.Files.newByteChannel(Files.java:432)
at java.base/java.nio.file.Files.readAllBytes(Files.java:3281)
at io.airbyte.commons.storage.StorageClientKt.gcsClient(StorageClient.kt:544)
at io.airbyte.commons.storage.GcsStorageClient.(StorageClient.kt:272)
at io.airbyte.commons.storage.StorageClientFactory.create(StorageClient.kt:72)
at io.airbyte.connectorSidecar.config.ApplicationFactory.workloadStorageClient(ApplicationFactory.kt:18)
```
---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/11409
Guida per i contributori
Apri la guida per i contributori
Valutazione
Questa issue non è ancora stata valutata.