dbt-labs / dbt-labs/dbt-cloud-openapi-spec

API v3: Credentials endpoint requires undocumented credential_details payload for BigQuery

Open
#79 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
12
Forks
10
PR merge metrics
No merged PRs in 30d

Description

## Summary

The v3 Admin API `Create Credentials` endpoint for BigQuery adapter credentials requires a nested `credential_details` payload that is **not documented** in the current OpenAPI spec or examples.

As a result, customers who follow the published schema cannot create working BigQuery v1 deployment credentials via API without reverse‑engineering browser calls.

---

## Affected endpoint

- `POST /api/v3/accounts/{account_id}/projects/{project_id}/credentials/`
- `PATCH /api/v3/accounts/{account_id}/projects/{project_id}/credentials/{id}/`

When creating **BigQuery v1 deployment credentials** with:

- `type: "adapter"`
- BigQuery v1 adapter

the API expects a `credential_details` object with:

- `adapter_version: "bigquery_v1"`
- `fields.*.metadata` / `fields.*.value` entries for:
- `auth_type`
- `schema`, `target_name`, `threads`
- BigQuery service account JSON fields (`project_id`, `private_key_id`, `private_key`, `client_email`, `client_id`, `auth_uri`, `token_uri`, `auth_provider_x509_cert_url`, `client_x509_cert_url`, etc.)

This structure is generated by the dbt Cloud UI but is not described anywhere in the OpenAPI spec.

---

## Current behavior

- The v3 OpenAPI spec for the Credentials endpoints does **not** describe `credential_details` or its nested fields for adapter credentials.
- If a customer follows the documented schema and omits `credential_details`, they can:
- End up with credentials objects that look valid in the UI but fail at runtime with:
- `Failed to generate profile due to incorrect credentials` (e.g. when `private_key.value` is empty).
- If a customer guesses and sends `credential_details` to the wrong endpoint or with the wrong type, they get:
- 400s like:
`Additional properties are not allowed ('credential_details' was unexpected)`
- The only reliable way to discover the required payload today is to:
1. Create credentials via the UI.
2. Inspect the network calls / `GET /credentials/{id}/` response.
3. Copy that structure into API scripts.

---

## Expected behavior

- The v3 OpenAPI spec and generated docs should:
- Clearly document that **adapter credentials** for warehouses like BigQuery v1 use `type: "adapter"` plus a `credential_details` object.
- Include the expected schema for `credential_details` (at least for BigQuery v1):
- `adapter_version: "bigquery_v1"`
- `fields..metadata`
- `fields..value`
- Provide at least one **example** payload for:
- BigQuery v1 deployment credentials using service account JSON.

This would allow customers to build automation purely from the published spec, without having to reverse‑engineer UI network calls.

---

## Customer impact

- Customer: **Agile Lab** ([Agile Lab ticket in Zendesk](https://dbtcloud.zendesk.com/agent/tickets/116538)).
- Goal: Use the v3 Admin API to automate project setup:
- One‑time creation of a BigQuery connection via the UI (uploading the service account JSON).
- Then programmatically create **deployment credentials and environments** via API only.
- Problems they hit:
- The OpenAPI spec does not mention `credential_details`, so they initially followed the documented schema and sent a simple body.
- In practice, the API requires a nested `credential_details.fields.*` payload (including `private_key`), which they had to infer from browser network calls.
- Incorrect or incomplete payloads produced:
- 400 responses: `Additional properties are not allowed ('credential_details' was unexpected)` on some calls.
- Runtime failures: `Failed to generate profile due to incorrect credentials` when `private_key.value` was empty.
- Credentials created via API appeared valid in the UI, but job runs only succeeded after manually editing & saving credentials in the UI.

Net effect: extra debugging effort, brittle automation, and confusion because the **effective API contract does not match the published spec or examples**.

---

## Requested changes

1. **Update `openapi-v3.yaml`** for the Credentials endpoints to:
- Add the `credential_details` object to the request body schema for adapter credentials.
- Document the nested structure for `fields.*.metadata` and `fields.*.value` for BigQuery v1 (and, ideally, other adapters that use this pattern)

2. **Add example payloads** to the v3 docs for:
- Creating BigQuery v1 deployment credentials via the Admin API, using service account JSON.
- Optionally, a short description in the docs page clarifying the intended flow:
- Connection via UI (or API),
- Adapter credentials with `credential_details`,
- Environment referencing `credentials_id`

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.