airbytehq / airbytehq/PyAirbyte

Unable to read data for source google sheets

オープン
#548 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る
主要言語
Python
スター
344
フォーク
77
平均マージ
1日 11時間
マージ済み PR(30日)
35

説明

The task is to sync data from google sheets to Postgres database. But facing read issues with google sheets.

Code :
```
import airbyte as ab
import json

source = ab.Source=ab.get_source("source-google-sheets")

account_string=json.dumps({
"type": "service_account",
"project_id": "test-443710",
"private_key_id": "20f0c1c481eb4b28aec6adc8b432960d14c54",
"private_key": "-----BEGIN PRIVATE KEY-----\nMIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQDtUTkajQwEl90n\nUb9zTZ+pxsHfBG2CBpkJgm4A4U3NhP6aqu8EtnVskqfzFKGuBdrshVMdYbR/\nwpNZCDGEMjx56/4KQGnSM0BdlKaAANod3rblhe/HYv9eL5VB+NAwVnjMkc2O9CaB\nBAMWlm+RDwKBgQCohWMZc9+nS8GHt7g/5EEjme9IG4ZSrPTPX/fmb91qGl8McEqnC6+G\nyOyVuH/cXqgnKJolbzanBroJTLmyEr66Fo9PLiiMwwKBgBbQO8lwSfZ/7Vm2ildE\nGo8vewzPBUFE6v75Y3KdKz+PLONiaf4fGx8YFcHDoJEfgATEoU/Ipf8xjd+Jmukd\nSGDvAodHLtSnoinCkUdA2McM8TBrxDrl/3Ir2oSvchNTkXOIx4osxqFo+C48MxMa\nkB0tyJvBHLMoboX/TUxG5efo\n-----END PRIVATE KEY-----\n",
"client_email": "test@test-443710.iam.gserviceaccount.com",
"client_id": "111345679374858679179",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://oauth2.googleapis.com/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/airbyte%40airbyte-443710.iam.gserviceaccount.com",
"universe_domain": "googleapis.com"
}
)

source.set_config(config={
"spreadsheet_id": "1MFrk0uOnoVrMvw7-QrfrwDk",
"credentials": {"auth_type":"Service","service_account_info":account_string}
})

source.check()

source.select_all_streams()
readResult:ab.ReadResult = source.read()
print("readresult",readResult)
userdf= readResult["users"].to_pandas()
print(userdf)
```
`source.check()` returns success.

But getting below error when try to read stream:
```
ERROR: Something went wrong in the connector. See the logs for more details.
───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Sync Progress: source-google-sheets -> DuckDBCache

───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Traceback (most recent call last):
File "/home/gwl/deepak/Learnings/PyAirbyte/first/main.py", line 46, in
readResult:ab.ReadResult = source.read()
File "/home/gwl/.local/lib/python3.10/site-packages/airbyte/sources/base.py", line 663, in read
result = self._read_to_cache(
File "/home/gwl/.local/lib/python3.10/site-packages/airbyte/sources/base.py", line 747, in _read_to_cache
cache._write_airbyte_message_stream( # noqa: SLF001 # Non-public API
File "/home/gwl/.local/lib/python3.10/site-packages/airbyte/caches/base.py", line 306, in _write_airbyte_message_stream
cache_processor.process_airbyte_messages(
File "/home/gwl/.local/lib/python3.10/site-packages/airbyte/shared/sql_processor.py", line 268, in process_airbyte_messages
for message in messages:
File "/home/gwl/.local/lib/python3.10/site-packages/airbyte/sources/base.py", line 576, in _read_with_catalog
yield from progress_tracker.tally_records_read(message_generator)
File "/home/gwl/.local/lib/python3.10/site-packages/airbyte/progress.py", line 261, in tally_records_read
for count, message in enumerate(messages, start=1):
File "/home/gwl/.local/lib/python3.10/site-packages/airbyte/_connector_base.py", line 453, in _execute
raise exc.AirbyteConnectorFailedError(
airbyte.exceptions.AirbyteConnectorFailedError: Connector failed. (AirbyteConnectorFailedError)
------------------------------------------------------------
AirbyteConnectorFailedError: Connector failed.
Please review the log file for more information.
Connector Name: 'source-google-sheets'
Exit Code: 1
Log file: /tmp/airbyte/logs/source-google-sheets/source-google-sheets-log-JE8PJAJSY.log
```

.log file:
```
2024-12-04 16:55:03 - INFO - Check succeeded
2024-12-04 16:55:05 - INFO - Running discovery on sheet 1MFrk0uOV6UFJ1F9kO0no-LXT3fKnfrwDk
2024-12-04 16:55:09 - INFO - Finished syncing spreadsheet 1MFrk0uOV6UFJ1F9kO0w7-QrfrwDk
2024-12-04 16:55:09 - INFO - ConnectorStateManager.__init__() got an unexpected keyword argument 'stream_instance_map'
Traceback (most recent call last):
File "/home/gwl/deepak/Learnings/PyAirbyte/first/.venv-source-google-sheets/bin/source-google-sheets", line 8, in
sys.exit(run())
File "/home/gwl/deepak/Learnings/PyAirbyte/first/.venv-source-google-sheets/lib/python3.10/site-packages/source_google_sheets/run.py", line 15, in run
launch(source, sys.argv[1:])
File "/home/gwl/deepak/Learnings/PyAirbyte/first/.venv-source-google-sheets/lib/python3.10/site-packages/airbyte_cdk/entrypoint.py", line 234, in launch
for message in source_entrypoint.run(parsed_args):
File "/home/gwl/deepak/Learnings/PyAirbyte/first/.venv-source-google-sheets/lib/python3.10/site-packages/airbyte_cdk/entrypoint.py", line 122, in run
yield from map(AirbyteEntrypoint.airbyte_message_to_string, self.read(source_spec, config, config_catalog, state))
File "/home/gwl/deepak/Learnings/PyAirbyte/first/.venv-source-google-sheets/lib/python3.10/site-packages/airbyte_cdk/entrypoint.py", line 164, in read
for message in self.source.read(self.logger, config, catalog, state):
File "/home/gwl/deepak/Learnings/PyAirbyte/first/.venv-source-google-sheets/lib/python3.10/site-packages/source_google_sheets/source.py", line 244, in read
yield from self._read(logger, config, catalog, state)
File "/home/gwl/deepak/Learnings/PyAirbyte/first/.venv-source-google-sheets/lib/python3.10/site-packages/source_google_sheets/source.py", line 166, in _read
state_manager = ConnectorStateManager(stream_instance_map=stream_instances, state=state or {})
TypeError: ConnectorStateManager.__init__() got an unexpected keyword argument 'stream_instance_map'
2024-12-04 16:55:09 - ERROR - Something went wrong in the connector. See the logs for more details.
2024-12-04 16:55:09 - INFO - {"type":"TRACE","trace":{"type":"ERROR","emitted_at":1733311509882.466,"error":{"message":"Something went wrong in the connector. See the logs for more details.","internal_message":"ConnectorStateManager.__init__() got an unexpected keyword argument 'stream_instance_map'","stack_trace":"Traceback (most recent call last):\n File \"/home/gwl/deepak/Learnings/PyAirbyte/first/.venv-source-google-sheets/bin/source-google-sheets\", line 8, in \n sys.exit(run())\n File \"/home/gwl/deepak/Learnings/PyAirbyte/first/.venv-source-google-sheets/lib/python3.10/site-packages/source_google_sheets/run.py\", line 15, in run\n launch(source, sys.argv[1:])\n File \"/home/gwl/deepak/Learnings/PyAirbyte/first/.venv-source-google-sheets/lib/python3.10/site-packages/airbyte_cdk/entrypoint.py\", line 234, in launch\n for message in source_entrypoint.run(parsed_args):\n File \"/home/gwl/deepak/Learnings/PyAirbyte/first/.venv-source-google-sheets/lib/python3.10/site-packages/airbyte_cdk/entrypoint.py\", line 122, in run\n yield from map(AirbyteEntrypoint.airbyte_message_to_string, self.read(source_spec, config, config_catalog, state))\n File \"/home/gwl/deepak/Learnings/PyAirbyte/first/.venv-source-google-sheets/lib/python3.10/site-packages/airbyte_cdk/entrypoint.py\", line 164, in read\n for message in self.source.read(self.logger, config, catalog, state):\n File \"/home/gwl/deepak/Learnings/PyAirbyte/first/.venv-source-google-sheets/lib/python3.10/site-packages/source_google_sheets/source.py\", line 244, in read\n yield from self._read(logger, config, catalog, state)\n File \"/home/gwl/deepak/Learnings/PyAirbyte/first/.venv-source-google-sheets/lib/python3.10/site-packages/source_google_sheets/source.py\", line 166, in _read\n state_manager = ConnectorStateManager(stream_instance_map=stream_instances, state=state or {})\nTypeError: ConnectorStateManager.__init__() got an unexpected keyword argument 'stream_instance_map'\n","failure_type":"system_error","stream_descriptor":null}}}
```

コントリビューションガイド

コントリビューションガイドを開く

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。