airbytehq / airbytehq/PyAirbyte

Unable to read data for source google sheets

Aperta
#548 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
Python
Stelle
344
Fork
77
Merge medio
1g 11h
PR unite (30g)
35

Descrizione

The task is to sync data from google sheets to Postgres database. But facing read issues with google sheets.

Code :
```
import airbyte as ab
import json

source = ab.Source=ab.get_source("source-google-sheets")

account_string=json.dumps({
"type": "service_account",
"project_id": "test-443710",
"private_key_id": "20f0c1c481eb4b28aec6adc8b432960d14c54",
"private_key": "-----BEGIN PRIVATE KEY-----\nMIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQDtUTkajQwEl90n\nUb9zTZ+pxsHfBG2CBpkJgm4A4U3NhP6aqu8EtnVskqfzFKGuBdrshVMdYbR/\nwpNZCDGEMjx56/4KQGnSM0BdlKaAANod3rblhe/HYv9eL5VB+NAwVnjMkc2O9CaB\nBAMWlm+RDwKBgQCohWMZc9+nS8GHt7g/5EEjme9IG4ZSrPTPX/fmb91qGl8McEqnC6+G\nyOyVuH/cXqgnKJolbzanBroJTLmyEr66Fo9PLiiMwwKBgBbQO8lwSfZ/7Vm2ildE\nGo8vewzPBUFE6v75Y3KdKz+PLONiaf4fGx8YFcHDoJEfgATEoU/Ipf8xjd+Jmukd\nSGDvAodHLtSnoinCkUdA2McM8TBrxDrl/3Ir2oSvchNTkXOIx4osxqFo+C48MxMa\nkB0tyJvBHLMoboX/TUxG5efo\n-----END PRIVATE KEY-----\n",
"client_email": "test@test-443710.iam.gserviceaccount.com",
"client_id": "111345679374858679179",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://oauth2.googleapis.com/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/airbyte%40airbyte-443710.iam.gserviceaccount.com",
"universe_domain": "googleapis.com"
}
)

source.set_config(config={
"spreadsheet_id": "1MFrk0uOnoVrMvw7-QrfrwDk",
"credentials": {"auth_type":"Service","service_account_info":account_string}
})

source.check()

source.select_all_streams()
readResult:ab.ReadResult = source.read()
print("readresult",readResult)
userdf= readResult["users"].to_pandas()
print(userdf)
```
`source.check()` returns success.

But getting below error when try to read stream:
```
ERROR: Something went wrong in the connector. See the logs for more details.
───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Sync Progress: source-google-sheets -> DuckDBCache

───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Traceback (most recent call last):
File "/home/gwl/deepak/Learnings/PyAirbyte/first/main.py", line 46, in
readResult:ab.ReadResult = source.read()
File "/home/gwl/.local/lib/python3.10/site-packages/airbyte/sources/base.py", line 663, in read
result = self._read_to_cache(
File "/home/gwl/.local/lib/python3.10/site-packages/airbyte/sources/base.py", line 747, in _read_to_cache
cache._write_airbyte_message_stream( # noqa: SLF001 # Non-public API
File "/home/gwl/.local/lib/python3.10/site-packages/airbyte/caches/base.py", line 306, in _write_airbyte_message_stream
cache_processor.process_airbyte_messages(
File "/home/gwl/.local/lib/python3.10/site-packages/airbyte/shared/sql_processor.py", line 268, in process_airbyte_messages
for message in messages:
File "/home/gwl/.local/lib/python3.10/site-packages/airbyte/sources/base.py", line 576, in _read_with_catalog
yield from progress_tracker.tally_records_read(message_generator)
File "/home/gwl/.local/lib/python3.10/site-packages/airbyte/progress.py", line 261, in tally_records_read
for count, message in enumerate(messages, start=1):
File "/home/gwl/.local/lib/python3.10/site-packages/airbyte/_connector_base.py", line 453, in _execute
raise exc.AirbyteConnectorFailedError(
airbyte.exceptions.AirbyteConnectorFailedError: Connector failed. (AirbyteConnectorFailedError)
------------------------------------------------------------
AirbyteConnectorFailedError: Connector failed.
Please review the log file for more information.
Connector Name: 'source-google-sheets'
Exit Code: 1
Log file: /tmp/airbyte/logs/source-google-sheets/source-google-sheets-log-JE8PJAJSY.log
```

.log file:
```
2024-12-04 16:55:03 - INFO - Check succeeded
2024-12-04 16:55:05 - INFO - Running discovery on sheet 1MFrk0uOV6UFJ1F9kO0no-LXT3fKnfrwDk
2024-12-04 16:55:09 - INFO - Finished syncing spreadsheet 1MFrk0uOV6UFJ1F9kO0w7-QrfrwDk
2024-12-04 16:55:09 - INFO - ConnectorStateManager.__init__() got an unexpected keyword argument 'stream_instance_map'
Traceback (most recent call last):
File "/home/gwl/deepak/Learnings/PyAirbyte/first/.venv-source-google-sheets/bin/source-google-sheets", line 8, in
sys.exit(run())
File "/home/gwl/deepak/Learnings/PyAirbyte/first/.venv-source-google-sheets/lib/python3.10/site-packages/source_google_sheets/run.py", line 15, in run
launch(source, sys.argv[1:])
File "/home/gwl/deepak/Learnings/PyAirbyte/first/.venv-source-google-sheets/lib/python3.10/site-packages/airbyte_cdk/entrypoint.py", line 234, in launch
for message in source_entrypoint.run(parsed_args):
File "/home/gwl/deepak/Learnings/PyAirbyte/first/.venv-source-google-sheets/lib/python3.10/site-packages/airbyte_cdk/entrypoint.py", line 122, in run
yield from map(AirbyteEntrypoint.airbyte_message_to_string, self.read(source_spec, config, config_catalog, state))
File "/home/gwl/deepak/Learnings/PyAirbyte/first/.venv-source-google-sheets/lib/python3.10/site-packages/airbyte_cdk/entrypoint.py", line 164, in read
for message in self.source.read(self.logger, config, catalog, state):
File "/home/gwl/deepak/Learnings/PyAirbyte/first/.venv-source-google-sheets/lib/python3.10/site-packages/source_google_sheets/source.py", line 244, in read
yield from self._read(logger, config, catalog, state)
File "/home/gwl/deepak/Learnings/PyAirbyte/first/.venv-source-google-sheets/lib/python3.10/site-packages/source_google_sheets/source.py", line 166, in _read
state_manager = ConnectorStateManager(stream_instance_map=stream_instances, state=state or {})
TypeError: ConnectorStateManager.__init__() got an unexpected keyword argument 'stream_instance_map'
2024-12-04 16:55:09 - ERROR - Something went wrong in the connector. See the logs for more details.
2024-12-04 16:55:09 - INFO - {"type":"TRACE","trace":{"type":"ERROR","emitted_at":1733311509882.466,"error":{"message":"Something went wrong in the connector. See the logs for more details.","internal_message":"ConnectorStateManager.__init__() got an unexpected keyword argument 'stream_instance_map'","stack_trace":"Traceback (most recent call last):\n File \"/home/gwl/deepak/Learnings/PyAirbyte/first/.venv-source-google-sheets/bin/source-google-sheets\", line 8, in \n sys.exit(run())\n File \"/home/gwl/deepak/Learnings/PyAirbyte/first/.venv-source-google-sheets/lib/python3.10/site-packages/source_google_sheets/run.py\", line 15, in run\n launch(source, sys.argv[1:])\n File \"/home/gwl/deepak/Learnings/PyAirbyte/first/.venv-source-google-sheets/lib/python3.10/site-packages/airbyte_cdk/entrypoint.py\", line 234, in launch\n for message in source_entrypoint.run(parsed_args):\n File \"/home/gwl/deepak/Learnings/PyAirbyte/first/.venv-source-google-sheets/lib/python3.10/site-packages/airbyte_cdk/entrypoint.py\", line 122, in run\n yield from map(AirbyteEntrypoint.airbyte_message_to_string, self.read(source_spec, config, config_catalog, state))\n File \"/home/gwl/deepak/Learnings/PyAirbyte/first/.venv-source-google-sheets/lib/python3.10/site-packages/airbyte_cdk/entrypoint.py\", line 164, in read\n for message in self.source.read(self.logger, config, catalog, state):\n File \"/home/gwl/deepak/Learnings/PyAirbyte/first/.venv-source-google-sheets/lib/python3.10/site-packages/source_google_sheets/source.py\", line 244, in read\n yield from self._read(logger, config, catalog, state)\n File \"/home/gwl/deepak/Learnings/PyAirbyte/first/.venv-source-google-sheets/lib/python3.10/site-packages/source_google_sheets/source.py\", line 166, in _read\n state_manager = ConnectorStateManager(stream_instance_map=stream_instances, state=state or {})\nTypeError: ConnectorStateManager.__init__() got an unexpected keyword argument 'stream_instance_map'\n","failure_type":"system_error","stream_descriptor":null}}}
```

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.