airbytehq / airbytehq/PyAirbyte
Unable to read data for source google sheets
- Linguagem predominante
- Python
- Estrelas
- 344
- Forks
- 77
- Merge médio
- 1d 11h
- PRs com merge (30d)
- 35
Descrição
The task is to sync data from google sheets to Postgres database. But facing read issues with google sheets.
Code :
```
import airbyte as ab
import json
source = ab.Source=ab.get_source("source-google-sheets")
account_string=json.dumps({
"type": "service_account",
"project_id": "test-443710",
"private_key_id": "20f0c1c481eb4b28aec6adc8b432960d14c54",
"private_key": "-----BEGIN PRIVATE KEY-----\nMIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQDtUTkajQwEl90n\nUb9zTZ+pxsHfBG2CBpkJgm4A4U3NhP6aqu8EtnVskqfzFKGuBdrshVMdYbR/\nwpNZCDGEMjx56/4KQGnSM0BdlKaAANod3rblhe/HYv9eL5VB+NAwVnjMkc2O9CaB\nBAMWlm+RDwKBgQCohWMZc9+nS8GHt7g/5EEjme9IG4ZSrPTPX/fmb91qGl8McEqnC6+G\nyOyVuH/cXqgnKJolbzanBroJTLmyEr66Fo9PLiiMwwKBgBbQO8lwSfZ/7Vm2ildE\nGo8vewzPBUFE6v75Y3KdKz+PLONiaf4fGx8YFcHDoJEfgATEoU/Ipf8xjd+Jmukd\nSGDvAodHLtSnoinCkUdA2McM8TBrxDrl/3Ir2oSvchNTkXOIx4osxqFo+C48MxMa\nkB0tyJvBHLMoboX/TUxG5efo\n-----END PRIVATE KEY-----\n",
"client_email": "test@test-443710.iam.gserviceaccount.com",
"client_id": "111345679374858679179",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://oauth2.googleapis.com/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/airbyte%40airbyte-443710.iam.gserviceaccount.com",
"universe_domain": "googleapis.com"
}
)
source.set_config(config={
"spreadsheet_id": "1MFrk0uOnoVrMvw7-QrfrwDk",
"credentials": {"auth_type":"Service","service_account_info":account_string}
})
source.check()
source.select_all_streams()
readResult:ab.ReadResult = source.read()
print("readresult",readResult)
userdf= readResult["users"].to_pandas()
print(userdf)
```
`source.check()` returns success.
But getting below error when try to read stream:
```
ERROR: Something went wrong in the connector. See the logs for more details.
───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Sync Progress: source-google-sheets -> DuckDBCache
───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Traceback (most recent call last):
File "/home/gwl/deepak/Learnings/PyAirbyte/first/main.py", line 46, in
readResult:ab.ReadResult = source.read()
File "/home/gwl/.local/lib/python3.10/site-packages/airbyte/sources/base.py", line 663, in read
result = self._read_to_cache(
File "/home/gwl/.local/lib/python3.10/site-packages/airbyte/sources/base.py", line 747, in _read_to_cache
cache._write_airbyte_message_stream( # noqa: SLF001 # Non-public API
File "/home/gwl/.local/lib/python3.10/site-packages/airbyte/caches/base.py", line 306, in _write_airbyte_message_stream
cache_processor.process_airbyte_messages(
File "/home/gwl/.local/lib/python3.10/site-packages/airbyte/shared/sql_processor.py", line 268, in process_airbyte_messages
for message in messages:
File "/home/gwl/.local/lib/python3.10/site-packages/airbyte/sources/base.py", line 576, in _read_with_catalog
yield from progress_tracker.tally_records_read(message_generator)
File "/home/gwl/.local/lib/python3.10/site-packages/airbyte/progress.py", line 261, in tally_records_read
for count, message in enumerate(messages, start=1):
File "/home/gwl/.local/lib/python3.10/site-packages/airbyte/_connector_base.py", line 453, in _execute
raise exc.AirbyteConnectorFailedError(
airbyte.exceptions.AirbyteConnectorFailedError: Connector failed. (AirbyteConnectorFailedError)
------------------------------------------------------------
AirbyteConnectorFailedError: Connector failed.
Please review the log file for more information.
Connector Name: 'source-google-sheets'
Exit Code: 1
Log file: /tmp/airbyte/logs/source-google-sheets/source-google-sheets-log-JE8PJAJSY.log
```
.log file:
```
2024-12-04 16:55:03 - INFO - Check succeeded
2024-12-04 16:55:05 - INFO - Running discovery on sheet 1MFrk0uOV6UFJ1F9kO0no-LXT3fKnfrwDk
2024-12-04 16:55:09 - INFO - Finished syncing spreadsheet 1MFrk0uOV6UFJ1F9kO0w7-QrfrwDk
2024-12-04 16:55:09 - INFO - ConnectorStateManager.__init__() got an unexpected keyword argument 'stream_instance_map'
Traceback (most recent call last):
File "/home/gwl/deepak/Learnings/PyAirbyte/first/.venv-source-google-sheets/bin/source-google-sheets", line 8, in
sys.exit(run())
File "/home/gwl/deepak/Learnings/PyAirbyte/first/.venv-source-google-sheets/lib/python3.10/site-packages/source_google_sheets/run.py", line 15, in run
launch(source, sys.argv[1:])
File "/home/gwl/deepak/Learnings/PyAirbyte/first/.venv-source-google-sheets/lib/python3.10/site-packages/airbyte_cdk/entrypoint.py", line 234, in launch
for message in source_entrypoint.run(parsed_args):
File "/home/gwl/deepak/Learnings/PyAirbyte/first/.venv-source-google-sheets/lib/python3.10/site-packages/airbyte_cdk/entrypoint.py", line 122, in run
yield from map(AirbyteEntrypoint.airbyte_message_to_string, self.read(source_spec, config, config_catalog, state))
File "/home/gwl/deepak/Learnings/PyAirbyte/first/.venv-source-google-sheets/lib/python3.10/site-packages/airbyte_cdk/entrypoint.py", line 164, in read
for message in self.source.read(self.logger, config, catalog, state):
File "/home/gwl/deepak/Learnings/PyAirbyte/first/.venv-source-google-sheets/lib/python3.10/site-packages/source_google_sheets/source.py", line 244, in read
yield from self._read(logger, config, catalog, state)
File "/home/gwl/deepak/Learnings/PyAirbyte/first/.venv-source-google-sheets/lib/python3.10/site-packages/source_google_sheets/source.py", line 166, in _read
state_manager = ConnectorStateManager(stream_instance_map=stream_instances, state=state or {})
TypeError: ConnectorStateManager.__init__() got an unexpected keyword argument 'stream_instance_map'
2024-12-04 16:55:09 - ERROR - Something went wrong in the connector. See the logs for more details.
2024-12-04 16:55:09 - INFO - {"type":"TRACE","trace":{"type":"ERROR","emitted_at":1733311509882.466,"error":{"message":"Something went wrong in the connector. See the logs for more details.","internal_message":"ConnectorStateManager.__init__() got an unexpected keyword argument 'stream_instance_map'","stack_trace":"Traceback (most recent call last):\n File \"/home/gwl/deepak/Learnings/PyAirbyte/first/.venv-source-google-sheets/bin/source-google-sheets\", line 8, in \n sys.exit(run())\n File \"/home/gwl/deepak/Learnings/PyAirbyte/first/.venv-source-google-sheets/lib/python3.10/site-packages/source_google_sheets/run.py\", line 15, in run\n launch(source, sys.argv[1:])\n File \"/home/gwl/deepak/Learnings/PyAirbyte/first/.venv-source-google-sheets/lib/python3.10/site-packages/airbyte_cdk/entrypoint.py\", line 234, in launch\n for message in source_entrypoint.run(parsed_args):\n File \"/home/gwl/deepak/Learnings/PyAirbyte/first/.venv-source-google-sheets/lib/python3.10/site-packages/airbyte_cdk/entrypoint.py\", line 122, in run\n yield from map(AirbyteEntrypoint.airbyte_message_to_string, self.read(source_spec, config, config_catalog, state))\n File \"/home/gwl/deepak/Learnings/PyAirbyte/first/.venv-source-google-sheets/lib/python3.10/site-packages/airbyte_cdk/entrypoint.py\", line 164, in read\n for message in self.source.read(self.logger, config, catalog, state):\n File \"/home/gwl/deepak/Learnings/PyAirbyte/first/.venv-source-google-sheets/lib/python3.10/site-packages/source_google_sheets/source.py\", line 244, in read\n yield from self._read(logger, config, catalog, state)\n File \"/home/gwl/deepak/Learnings/PyAirbyte/first/.venv-source-google-sheets/lib/python3.10/site-packages/source_google_sheets/source.py\", line 166, in _read\n state_manager = ConnectorStateManager(stream_instance_map=stream_instances, state=state or {})\nTypeError: ConnectorStateManager.__init__() got an unexpected keyword argument 'stream_instance_map'\n","failure_type":"system_error","stream_descriptor":null}}}
```
Guia de contribuição
Avaliação
Esta issue ainda não foi avaliada.