dlt-hub / dlt-hub/dlt

dlt tries OAuth in GCP Instead of Service Account Credentials when given Service Account Credentials via credential file

Open
#3,328 2 comments 1 reaction 0 assignees View on GitHub
good first issue
Dominant language
Python
Stars
5.9k
Forks
600
Avg merge
1d 14h
Merged PRs (30d)
38

Description

### dlt version

dlt 1.17.1

### Describe the problem

When using GCP service account credentials, referenced either in environment variables or in `secrets.toml` I get this following error.

```
`GcpOAuthCredentials` cannot parse the configuration value provided. The value is of type `str` and comes from the sections `('credentials',)` Value may be a secret and is not shown. Details: The expected representation for `GcpOAuthCredentials` is a string with serialized oauth2 user info and may be wrapped in 'install'/'web' node - depending of oauth2 app type.
```

### Expected behavior

dlt should be using `GcpServiceAccountCredentials`, which currently can be used when explicitly provided within the pipeline as in

```
with open("creds.json", "r") as f:
creds = json.load(f)

service_dict = json.loads(json.dumps(creds))

gcp_credentials = GcpServiceAccountCredentials()
gcp_credentials.parse_native_representation(json.dumps(creds))
```

### Steps to reproduce

set up GCP service account credentials and attempt to use them either in secrets.toml as in

```
[destination.filesystem]
bucket_url = "gs://yourbucket...

[destination.filesystem]
credentials="path/to/your/json"
```

when running a dlt pipeline for filesystem destination

### Operating system

macOS

### Runtime environment

Local

### Python version

3.10

### dlt data source

API

### dlt destination

Filesystem & buckets

### Other deployment details

_No response_

### Additional information

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by tracing how the filesystem destination resolves credentials from environment variables and secrets.toml, comparing GcpOAuthCredentials with GcpServiceAccountCredentials. Reproduce the reported pipeline with a service-account JSON credential file and confirm that the file is accepted without explicit credential parsing in the pipeline.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, python
Domain
cloud
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.