airbytehq / airbytehq/PyAirbyte

Unable to read data for source google sheets

Aberta
#548 0 comentários 0 reações 0 responsáveis Ver no GitHub
Linguagem predominante
Python
Estrelas
344
Forks
77
Merge médio
1d 11h
PRs com merge (30d)
35

Descrição

The task is to sync data from google sheets to Postgres database. But facing read issues with google sheets.

Code :
```
import airbyte as ab
import json

source = ab.Source=ab.get_source("source-google-sheets")

account_string=json.dumps({
"type": "service_account",
"project_id": "test-443710",
"private_key_id": "20f0c1c481eb4b28aec6adc8b432960d14c54",
"private_key": "-----BEGIN PRIVATE KEY-----\nMIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQDtUTkajQwEl90n\nUb9zTZ+pxsHfBG2CBpkJgm4A4U3NhP6aqu8EtnVskqfzFKGuBdrshVMdYbR/\nwpNZCDGEMjx56/4KQGnSM0BdlKaAANod3rblhe/HYv9eL5VB+NAwVnjMkc2O9CaB\nBAMWlm+RDwKBgQCohWMZc9+nS8GHt7g/5EEjme9IG4ZSrPTPX/fmb91qGl8McEqnC6+G\nyOyVuH/cXqgnKJolbzanBroJTLmyEr66Fo9PLiiMwwKBgBbQO8lwSfZ/7Vm2ildE\nGo8vewzPBUFE6v75Y3KdKz+PLONiaf4fGx8YFcHDoJEfgATEoU/Ipf8xjd+Jmukd\nSGDvAodHLtSnoinCkUdA2McM8TBrxDrl/3Ir2oSvchNTkXOIx4osxqFo+C48MxMa\nkB0tyJvBHLMoboX/TUxG5efo\n-----END PRIVATE KEY-----\n",
"client_email": "test@test-443710.iam.gserviceaccount.com",
"client_id": "111345679374858679179",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://oauth2.googleapis.com/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/airbyte%40airbyte-443710.iam.gserviceaccount.com",
"universe_domain": "googleapis.com"
}
)

source.set_config(config={
"spreadsheet_id": "1MFrk0uOnoVrMvw7-QrfrwDk",
"credentials": {"auth_type":"Service","service_account_info":account_string}
})

source.check()

source.select_all_streams()
readResult:ab.ReadResult = source.read()
print("readresult",readResult)
userdf= readResult["users"].to_pandas()
print(userdf)
```
`source.check()` returns success.

But getting below error when try to read stream:
```
ERROR: Something went wrong in the connector. See the logs for more details.
───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Sync Progress: source-google-sheets -> DuckDBCache

───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Traceback (most recent call last):
File "/home/gwl/deepak/Learnings/PyAirbyte/first/main.py", line 46, in
readResult:ab.ReadResult = source.read()
File "/home/gwl/.local/lib/python3.10/site-packages/airbyte/sources/base.py", line 663, in read
result = self._read_to_cache(
File "/home/gwl/.local/lib/python3.10/site-packages/airbyte/sources/base.py", line 747, in _read_to_cache
cache._write_airbyte_message_stream( # noqa: SLF001 # Non-public API
File "/home/gwl/.local/lib/python3.10/site-packages/airbyte/caches/base.py", line 306, in _write_airbyte_message_stream
cache_processor.process_airbyte_messages(
File "/home/gwl/.local/lib/python3.10/site-packages/airbyte/shared/sql_processor.py", line 268, in process_airbyte_messages
for message in messages:
File "/home/gwl/.local/lib/python3.10/site-packages/airbyte/sources/base.py", line 576, in _read_with_catalog
yield from progress_tracker.tally_records_read(message_generator)
File "/home/gwl/.local/lib/python3.10/site-packages/airbyte/progress.py", line 261, in tally_records_read
for count, message in enumerate(messages, start=1):
File "/home/gwl/.local/lib/python3.10/site-packages/airbyte/_connector_base.py", line 453, in _execute
raise exc.AirbyteConnectorFailedError(
airbyte.exceptions.AirbyteConnectorFailedError: Connector failed. (AirbyteConnectorFailedError)
------------------------------------------------------------
AirbyteConnectorFailedError: Connector failed.
Please review the log file for more information.
Connector Name: 'source-google-sheets'
Exit Code: 1
Log file: /tmp/airbyte/logs/source-google-sheets/source-google-sheets-log-JE8PJAJSY.log
```

.log file:
```
2024-12-04 16:55:03 - INFO - Check succeeded
2024-12-04 16:55:05 - INFO - Running discovery on sheet 1MFrk0uOV6UFJ1F9kO0no-LXT3fKnfrwDk
2024-12-04 16:55:09 - INFO - Finished syncing spreadsheet 1MFrk0uOV6UFJ1F9kO0w7-QrfrwDk
2024-12-04 16:55:09 - INFO - ConnectorStateManager.__init__() got an unexpected keyword argument 'stream_instance_map'
Traceback (most recent call last):
File "/home/gwl/deepak/Learnings/PyAirbyte/first/.venv-source-google-sheets/bin/source-google-sheets", line 8, in
sys.exit(run())
File "/home/gwl/deepak/Learnings/PyAirbyte/first/.venv-source-google-sheets/lib/python3.10/site-packages/source_google_sheets/run.py", line 15, in run
launch(source, sys.argv[1:])
File "/home/gwl/deepak/Learnings/PyAirbyte/first/.venv-source-google-sheets/lib/python3.10/site-packages/airbyte_cdk/entrypoint.py", line 234, in launch
for message in source_entrypoint.run(parsed_args):
File "/home/gwl/deepak/Learnings/PyAirbyte/first/.venv-source-google-sheets/lib/python3.10/site-packages/airbyte_cdk/entrypoint.py", line 122, in run
yield from map(AirbyteEntrypoint.airbyte_message_to_string, self.read(source_spec, config, config_catalog, state))
File "/home/gwl/deepak/Learnings/PyAirbyte/first/.venv-source-google-sheets/lib/python3.10/site-packages/airbyte_cdk/entrypoint.py", line 164, in read
for message in self.source.read(self.logger, config, catalog, state):
File "/home/gwl/deepak/Learnings/PyAirbyte/first/.venv-source-google-sheets/lib/python3.10/site-packages/source_google_sheets/source.py", line 244, in read
yield from self._read(logger, config, catalog, state)
File "/home/gwl/deepak/Learnings/PyAirbyte/first/.venv-source-google-sheets/lib/python3.10/site-packages/source_google_sheets/source.py", line 166, in _read
state_manager = ConnectorStateManager(stream_instance_map=stream_instances, state=state or {})
TypeError: ConnectorStateManager.__init__() got an unexpected keyword argument 'stream_instance_map'
2024-12-04 16:55:09 - ERROR - Something went wrong in the connector. See the logs for more details.
2024-12-04 16:55:09 - INFO - {"type":"TRACE","trace":{"type":"ERROR","emitted_at":1733311509882.466,"error":{"message":"Something went wrong in the connector. See the logs for more details.","internal_message":"ConnectorStateManager.__init__() got an unexpected keyword argument 'stream_instance_map'","stack_trace":"Traceback (most recent call last):\n File \"/home/gwl/deepak/Learnings/PyAirbyte/first/.venv-source-google-sheets/bin/source-google-sheets\", line 8, in \n sys.exit(run())\n File \"/home/gwl/deepak/Learnings/PyAirbyte/first/.venv-source-google-sheets/lib/python3.10/site-packages/source_google_sheets/run.py\", line 15, in run\n launch(source, sys.argv[1:])\n File \"/home/gwl/deepak/Learnings/PyAirbyte/first/.venv-source-google-sheets/lib/python3.10/site-packages/airbyte_cdk/entrypoint.py\", line 234, in launch\n for message in source_entrypoint.run(parsed_args):\n File \"/home/gwl/deepak/Learnings/PyAirbyte/first/.venv-source-google-sheets/lib/python3.10/site-packages/airbyte_cdk/entrypoint.py\", line 122, in run\n yield from map(AirbyteEntrypoint.airbyte_message_to_string, self.read(source_spec, config, config_catalog, state))\n File \"/home/gwl/deepak/Learnings/PyAirbyte/first/.venv-source-google-sheets/lib/python3.10/site-packages/airbyte_cdk/entrypoint.py\", line 164, in read\n for message in self.source.read(self.logger, config, catalog, state):\n File \"/home/gwl/deepak/Learnings/PyAirbyte/first/.venv-source-google-sheets/lib/python3.10/site-packages/source_google_sheets/source.py\", line 244, in read\n yield from self._read(logger, config, catalog, state)\n File \"/home/gwl/deepak/Learnings/PyAirbyte/first/.venv-source-google-sheets/lib/python3.10/site-packages/source_google_sheets/source.py\", line 166, in _read\n state_manager = ConnectorStateManager(stream_instance_map=stream_instances, state=state or {})\nTypeError: ConnectorStateManager.__init__() got an unexpected keyword argument 'stream_instance_map'\n","failure_type":"system_error","stream_descriptor":null}}}
```

Guia de contribuição

Abrir o guia de contribuição

Avaliação

Esta issue ainda não foi avaliada.

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.