dlt tries OAuth in GCP Instead of Service Account Credentials when given Service Account Credentials via credential file
- Dominant language
- Python
- Stars
- 5.9k
- Forks
- 600
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 38
Description
### dlt version
dlt 1.17.1
### Describe the problem
When using GCP service account credentials, referenced either in environment variables or in `secrets.toml` I get this following error.
```
`GcpOAuthCredentials` cannot parse the configuration value provided. The value is of type `str` and comes from the sections `('credentials',)` Value may be a secret and is not shown. Details: The expected representation for `GcpOAuthCredentials` is a string with serialized oauth2 user info and may be wrapped in 'install'/'web' node - depending of oauth2 app type.
```
### Expected behavior
dlt should be using `GcpServiceAccountCredentials`, which currently can be used when explicitly provided within the pipeline as in
```
with open("creds.json", "r") as f:
creds = json.load(f)
service_dict = json.loads(json.dumps(creds))
gcp_credentials = GcpServiceAccountCredentials()
gcp_credentials.parse_native_representation(json.dumps(creds))
```
### Steps to reproduce
set up GCP service account credentials and attempt to use them either in secrets.toml as in
```
[destination.filesystem]
bucket_url = "gs://yourbucket...
[destination.filesystem]
credentials="path/to/your/json"
```
when running a dlt pipeline for filesystem destination
### Operating system
macOS
### Runtime environment
Local
### Python version
3.10
### dlt data source
API
### dlt destination
Filesystem & buckets
### Other deployment details
_No response_
### Additional information
_No response_
Contributor guide
Research direction
Start by tracing how the filesystem destination resolves credentials from environment variables and secrets.toml, comparing GcpOAuthCredentials with GcpServiceAccountCredentials. Reproduce the reported pipeline with a service-account JSON credential file and confirm that the file is accepted without explicit credential parsing in the pipeline.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud, python
- Domain
- cloud
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100