airbytehq / airbytehq/PyAirbyte

Bug: `advanced_auth` is being parsed even if the yaml predicate does not match

Aperta
#759 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
Python
Stelle
344
Fork
77
Merge medio
1g 11h
PR unite (30g)
35

Descrizione

As title states. Steps to reproduce are
1. Use the provided yaml below. Notice I declared `SelectiveAuthenticator` which we can choose to use token
2. Create a ticktick account & application credentials to test [following this doc](https://developer.ticktick.com/api#/openapi?id=getting-started). (You can use this site I created to retrieve token)
3. Use the following snippet and add your token in `TICKTICK_TOKEN`
4. Observe pyairbyte complains as shown in log

Interestingly, if I commented out the whole `advanced_auth` block in the yaml file then the source will run.

```python
import airbyte as ab
from pathlib import Path
from dotenv import load_dotenv; load_dotenv()
import os; os.chdir(Path(__file__).cwd())
import yaml

ticktick_yaml = Path('./ticktick_source.yaml')
config = {
"authorization": {
"auth_type": "Token",
"bearer_token": f"{os.getenv("TICKTICK_TOKEN")}"
}
}

source = ab.get_source(
name='ticktick',
config=config,
source_manifest=ticktick_yaml,
streams="*"
)

source.check()
```

custom connector yaml

```yaml

version: 6.48.15

type: DeclarativeSource

description: Source for the ticktick openapi endpoint at https://developer.ticktick.com/

check:
type: CheckStream
stream_names:
- projects

definitions:
oauth_authenticator:
type: OAuthAuthenticator
client_id: "{{ config.authorization.client_id }}"
client_secret: "{{ config.authorization.client_secret) }}"
grant_type: client_credentials
access_token_value: "{{ config.authorization.client_access_token }}"
scopes:
- "tasks:"
- read
bearer_token:
type: BearerAuthenticator
api_token: "{{ config.authorization.bearer_token }}"

base_requester:
type: HttpRequester
url_base: https://api.ticktick.com
authenticator:
type: SelectiveAuthenticator
authenticator_selection_path: ["authorization", "auth_type"]
authenticators:
Oauth: "#/definitions/oauth_authenticator"
Token: "#/definitions/bearer_token"
error_handler:
type: DefaultErrorHandler
streams:
tasks:
type: DeclarativeStream
name: tasks
retriever:
type: SimpleRetriever
decoder:
type: JsonDecoder
requester:
$ref: "#/definitions/base_requester"
path: /open/v1/project/{{ stream_partition['parent_id'] }}/data
http_method: GET
record_selector:
type: RecordSelector
extractor:
type: DpathExtractor
field_path:
- tasks
record_filter:
type: RecordFilter
condition: "{{ record }}"
partition_router:
type: SubstreamPartitionRouter
parent_stream_configs:
- type: ParentStreamConfig
parent_key: id
partition_field: parent_id
stream:
$ref: "#/definitions/streams/projects"
primary_key:
- id
schema_loader:
type: InlineSchemaLoader
schema:
$ref: "#/schemas/tasks"
projects:
type: DeclarativeStream
name: projects
retriever:
type: SimpleRetriever
decoder:
type: JsonDecoder
requester:
$ref: "#/definitions/base_requester"
path: /open/v1/project/
record_selector:
type: RecordSelector
extractor:
type: DpathExtractor
field_path: []
record_filter:
type: RecordFilter
condition: "{{ not record.closed }}"
schema_normalization: Default
primary_key:
- id
schema_loader:
type: InlineSchemaLoader
schema:
$ref: "#/schemas/projects"

streams:
- $ref: "#/definitions/streams/projects"
- $ref: "#/definitions/streams/tasks"

spec:
type: Spec
advanced_auth:
auth_flow_type: oauth2.0
predicate_key:
- authorization
- auth_type
predicate_value: Oauth
oauth_config_specification:
complete_oauth_output_specification:
required:
- access_token
properties:
access_token:
type: string
path_in_connector_config:
- authorization
- client_access_token
oauth_connector_input_specification:
scope: "tasks: read"
consent_url: >-
https://ticktick.com/oauth/authorize?scope={{scope_value |
urlEncode}}&client_id={{client_id_value}}&state={{state}}&redirect_uri={{redirect_uri_value}}&response_type=code
extract_output:
- access_token
access_token_url: https://ticktick.com/oauth/token
access_token_params:
code: "{{ auth_code_value }}"
scope: "{{ scope_value | urlEncode }}"
grant_type: authorization_code
redirect_uri: "{{ redirect_uri_value }}"
access_token_headers:
Content-Type: application/x-www-form-urlencoded
Authorization: >-
Basic {{ (client_id_value ~ ':' ~ client_secret_value) | b64encode
}}
complete_oauth_server_input_specification:
required:
- client_id
- client_secret
properties:
client_id:
type: string
client_secret:
type: string
complete_oauth_server_output_specification:
required:
- client_id
- client_secret
properties:
client_id:
type: string
path_in_connector_config:
- authorization
- client_id
client_secret:
type: string
path_in_connector_config:
- authorization
- client_secret
connection_specification:
type: object
$schema: http://json-schema.org/draft-07/schema#
properties:
authorization:
type: object
title: Authentication Type
oneOf:
- title: OAuth2
type: object
required:
- auth_type
- client_id
- client_secret
properties:
auth_type:
type: string
const: Oauth
order: 0
client_id:
title: Client ID
type: string
description: The client ID of your Ticktick application.
Read more here.
airbyte_secret: true
client_secret:
title: Client Secret
type: string
description: The client secret of of your Ticktick application.
application. Read more here.
airbyte_secret: true
client_access_token:
title: Access Token
type: string
description: Access token for making authenticated requests; filled after complete oauth2 flow.
airbyte_secret: true
- type: object
title: Bearer Token (from Oauth2)
required:
- auth_type
- bearer_token
properties:
auth_type:
type: string
const: Token
order: 0
bearer_token:
title: Bearer Token
type: string
description: Access token for making authenticated requests; filled after complete oauth2 flow.
airbyte_secret: true
additionalProperties: true

metadata:
assist: {}
testedStreams:
tasks:
hasRecords: true
streamHash: 614b09c66bbaf39cfd22bf6fbc3b5c8d45bbe19d
hasResponse: true
primaryKeysAreUnique: true
primaryKeysArePresent: true
responsesAreSuccessful: true
projects:
hasRecords: true
streamHash: f302a05535246c68b860f6b24d368272c73ddb28
hasResponse: true
primaryKeysAreUnique: true
primaryKeysArePresent: true
responsesAreSuccessful: true
autoImportSchema:
tasks: true
projects: true

schemas:
tasks:
type: object
$schema: http://json-schema.org/schema#
required:
- id
properties:
id:
type: string
desc:
type:
- string
- "null"
etag:
type:
- string
- "null"
kind:
type:
- string
- "null"
tags:
type:
- array
- "null"
items:
type:
- string
- "null"
items:
type:
- array
- "null"
items:
type:
- object
- "null"
properties:
id:
type:
- string
- "null"
title:
type:
- string
- "null"
status:
type:
- number
- "null"
isAllDay:
type:
- boolean
- "null"
timeZone:
type:
- string
- "null"
sortOrder:
type:
- number
- "null"
title:
type:
- string
- "null"
status:
type:
- number
- "null"
content:
type:
- string
- "null"
dueDate:
type:
- string
- "null"
columnId:
type:
- string
- "null"
isAllDay:
type:
- boolean
- "null"
priority:
type:
- number
- "null"
timeZone:
type:
- string
- "null"
projectId:
type:
- string
- "null"
sortOrder:
type:
- number
- "null"
startDate:
type:
- string
- "null"
repeatFlag:
type:
- string
- "null"
additionalProperties: true
projects:
type: object
$schema: http://json-schema.org/schema#
required:
- id
properties:
id:
type: string
kind:
type:
- string
- "null"
name:
type:
- string
- "null"
color:
type:
- string
- "null"
closed:
type:
- boolean
- "null"
groupId:
type:
- string
- "null"
viewMode:
type:
- string
- "null"
sortOrder:
type:
- number
- "null"
permission:
type:
- string
- "null"
additionalProperties: true

api_budget:
type: HTTPAPIBudget
policies:
- type: MovingWindowCallRatePolicy
rates:
- limit: 3
interval: PT1S
- limit: 60
interval: PT1M
matchers:
- method: GET
url_path_pattern: .*
status_codes_for_ratelimit_hit:
- 503
- 500
- 429

```

log

```log
Exception has occurred: AirbyteConnectorCheckFailedError (note: full exception trace is shown but execution is paused at: _run_module_as_main)
Connector check failed. (AirbyteConnectorCheckFailedError)
------------------------------------------------------------
AirbyteConnectorCheckFailedError: Connector check failed.
Please review the log file for more information.
Connector Name: 'ticktick'
------------------------------------------------------------
Caused by: Connector failed. (AirbyteConnectorFailedError)
------------------------------------------------------------
AirbyteConnectorFailedError: Connector failed.
Please review the log file for more information.
Connector Name: 'ticktick'
------------------------------------------------------------
Caused by: 'str' object has no attribute 'value'
Log file: /tmp/airbyte/logs/ticktick/ticktick-log-K397H8YD3.log
Log file: /tmp/airbyte/logs/ticktick/ticktick-log-K397H8YD3.log
File "/home/ubuntu/dev/ticktick_dbt/.venv/lib/python3.12/site-packages/airbyte/_connector_base.py", line 469, in _execute
try:

message: AirbyteMessage = AirbyteMessage.model_validate_json(json_data=line)

if progress_tracker and message.record:

stream_name = message.record.stream

progress_tracker.tally_bytes_read(

bytes_read=len(line) - envelope_size - len(stream_name),

stream_name=stream_name,

)

self._peek_airbyte_message(message)

yield message

except Exception:

# This is likely a log message, so log it as INFO.

self._print_info_message(line)
File "/home/ubuntu/dev/ticktick_dbt/.venv/lib/python3.12/site-packages/airbyte/_executors/declarative.py", line 115, in execute
yield from source_entrypoint.run(parsed_args)
File "/home/ubuntu/dev/ticktick_dbt/.venv/lib/python3.12/site-packages/airbyte_cdk/entrypoint.py", line 168, in run
source_spec: ConnectorSpecification = self.source.spec(self.logger)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/dev/ticktick_dbt/.venv/lib/python3.12/site-packages/airbyte_cdk/sources/declarative/manifest_declarative_source.py", line 409, in spec
self._spec_component.generate_spec() if self._spec_component else super().spec(logger)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/dev/ticktick_dbt/.venv/lib/python3.12/site-packages/airbyte_cdk/sources/declarative/spec/spec.py", line 57, in generate_spec
self.advanced_auth.auth_flow_type = self.advanced_auth.auth_flow_type.value # type: ignore # We know this is always assigned to an AuthFlow which has the auth_flow_type field
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'str' object has no attribute 'value'

During handling of the above exception, another exception occurred:

File "/home/ubuntu/dev/ticktick_dbt/.venv/lib/python3.12/site-packages/airbyte/_connector_base.py", line 338, in check
if msg.type == Type.CONNECTION_STATUS and msg.connectionStatus:

if msg.connectionStatus.status != Status.FAILED:

rich.print(

f"Connection check succeeded for `{self.name}`.",

file=sys.stderr,

)

log_connector_check_result(

name=self.name,

state=EventState.SUCCEEDED,

)

return

log_connector_check_result(

name=self.name,

state=EventState.FAILED,

)

# Give logs a chance to be flushed.

sleep(1)

raise exc.AirbyteConnectorCheckFailedError(

connector_name=self.name,

help_url=self.docs_url,

context={

"failure_reason": msg.connectionStatus.message,

},

)

raise exc.AirbyteConnectorCheckFailedError(
File "/home/ubuntu/dev/ticktick_dbt/.venv/lib/python3.12/site-packages/airbyte/_connector_base.py", line 496, in _execute
connector_name=self.name,

log_text=self._last_log_messages,

original_exception=e,

) from None
airbyte.exceptions.AirbyteConnectorFailedError: Connector failed. (AirbyteConnectorFailedError)
------------------------------------------------------------
AirbyteConnectorFailedError: Connector failed.
Please review the log file for more information.
Connector Name: 'ticktick'
------------------------------------------------------------
Caused by: 'str' object has no attribute 'value'
Log file: /tmp/airbyte/logs/ticktick/ticktick-log-K397H8YD3.log

During handling of the above exception, another exception occurred:

File "/home/ubuntu/dev/ticktick_dbt/.venv/lib/python3.12/site-packages/airbyte/_connector_base.py", line 370, in check
connector_name=self.name,

original_exception=ex,

) from None
File "/home/ubuntu/dev/ticktick_dbt/EL/main.py", line 26, in
source.check()
File "/usr/lib/python3.12/runpy.py", line 88, in _run_code
exec(code, run_globals)
File "/usr/lib/python3.12/runpy.py", line 198, in _run_module_as_main (Current frame)
return _run_code(code, main_globals, None,
airbyte.exceptions.AirbyteConnectorCheckFailedError: Connector check failed. (AirbyteConnectorCheckFailedError)
------------------------------------------------------------
AirbyteConnectorCheckFailedError: Connector check failed.
Please review the log file for more information.
Connector Name: 'ticktick'
------------------------------------------------------------
Caused by: Connector failed. (AirbyteConnectorFailedError)
------------------------------------------------------------
AirbyteConnectorFailedError: Connector failed.
Please review the log file for more information.
Connector Name: 'ticktick'
------------------------------------------------------------
Caused by: 'str' object has no attribute 'value'
Log file: /tmp/airbyte/logs/ticktick/ticktick-log-K397H8YD3.log
Log file: /tmp/airbyte/logs/ticktick/ticktick-log-K397H8YD3.log
```

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.