airbytehq / airbytehq/PyAirbyte

Pyairbyte destination-bigquery columns are empty

Abierto
#618 0 comentarios 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
Python
Estrellas
344
Forks
77
Merge medio
1 d 11 h
PR fusionados (30 d)
35

Descripción

Hi,

I'm reaching you because my bigquery destination is not working as expected.

1. Here is my bigquery cache result :
Image

2. Here the final dataset destination result
Image

As we can see, values are absents and metadata are not matching.
I'm on aibryte 0.23.0 and my code is the following:

```python
import json
import envs
import airbyte as ab
from airbyte.caches import BigQueryCache

service_account_credentials_path = "service-account-credentials.json"
user_credentials = {...}

with open(service_account_credentials_path, 'r') as file:
service_account_credentials_json = json.load(file)

site_urls = ["...hidden"]

def source_google_search_console(project_id: str, site_urls: list, user_credentials: dict):
source = ab.get_source(
"source-google-search-console",
#install_if_missing = True,
config={
"site_urls": site_urls,
"start_date": "2025-01-01",
"end_date": "2025-01-01",
"authorization": user_credentials
}
)

# Verify the config and creds by running `check`:
source.check()
source.select_streams(['sites'])
#source.select_all_streams()

cache = BigQueryCache(project_name=project_id, credentials_path=service_account_credentials_path, dataset_name="airbyte_cache")
result = source.read(cache, force_full_refresh=True)
return result, cache

def destination_bigquery(result, cache, project_id: str, dataset_id: str, region: str, credentials_json: str, destination_type: str ="destination-bigquery"):
destination = ab.get_destination(
destination_type,
config = {
"project_id": project_id,
"dataset_id": dataset_id,
"dataset_location": region,
"credentials_json": json.dumps(credentials_json)
}
)
destination.check()
destination.write(source_data=result, cache=cache)
return destination

# Calls
gsc_result, bq_cache = source_google_search_console(envs.PROJECT_ID, site_urls, user_credentials)

res = destination_bigquery(gsc_result, bq_cache, envs.PROJECT_ID, "pyairbyte_tests", "EU", service_account_credentials_json)
```

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.