airbytehq / airbytehq/PyAirbyte

Failure to run `spec` on `source-airtable` ("Error: Validation against json schema defined in declarative_component_schema.yaml schema failed")

Open
#671 2 comments 8 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
344
Forks
77
Avg merge
1d 11h
Merged PRs (30d)
35

Description

Reported in slack:

- https://airbytehq-team.slack.com/archives/C06FZ238P8W/p1747412543018439

The failure occurs when invoking `source-airtable`:

```
Error: Validation against json schema defined in declarative_component_schema.yaml schema failed.
```

Easy to repro:

```
uv tool install airbyte-source-airtable
source-airtable spec
```

Error:

Show/Hide

> `$ source-airtable spec`
> `{"type":"TRACE","trace":{"type":"ERROR","emitted_at":1747413222276,"error":{"message":"Error starting the sync. This could be due to an invalid configuration or catalog. Please contact Support for assistance. Error: Validation against json schema defined in declarative_component_schema.yaml schema failed","stack_trace":"Traceback (most recent call last):\n File \"/Users/ajsteers/.local/share/uv/tools/airbyte-source-airtable/lib/python3.12/site-packages/airbyte_cdk/sources/declarative/manifest_declarative_source.py\", line 394, in _validate_source\n validate(self._source_config, self._declarative_component_schema)\n File \"/Users/ajsteers/.local/share/uv/tools/airbyte-source-airtable/lib/python3.12/site-packages/jsonschema/validators.py\", line 1121, in validate\n raise error\njsonschema.exceptions.ValidationError: {'type': 'HttpComponentsResolver', 'retriever': {'type': 'SimpleRetriever', 'requester': {'type': 'HttpRequester', 'url_base': 'https://api.airtable.com/v0/', 'authenticator': {'type': 'SelectiveAuthenticator', 'authenticator_selection_path': ['credentials', 'auth_method'], 'authenticators': {'oauth2.0': {'type': 'OAuthAuthenticator', 'client_id': \"{{ config['credentials']['client_id'] }}\", 'client_secret': \"{{ config['credentials']['client_secret'] }}\", 'refresh_token': \"{{ config['credentials']['refresh_token'] }}\", 'grant_type': 'refresh_token', 'refresh_request_headers': {'Authorization': \"Basic {{ [config['credentials']['client_id'], config['credentials']['client_secret']] | join(':') | base64encode }}\", 'Content-Type': 'application/x-www-form-urlencoded'}, 'refresh_token_updater': {'refresh_token_error_status_codes': [400, 401], 'refresh_token_error_key': 'error', 'refresh_token_error_values': ['invalid_grant', 'invalid_client'], 'refresh_token_name': 'refresh_token', 'access_token_config_path': ['credentials', 'access_token'], 'token_expiry_date_config_path': ['credentials', 'token_expiry_date'], 'refresh_token_config_path': ['credentials', 'refresh_token']}, 'token_refresh_endpoint': 'https://airtable.com/oauth2/v1/token'}, 'api_key': {'type': 'BearerAuthenticator', 'api_token': \"{{ config['credentials']['api_key'] }}\"}}}, 'error_handler': {'type': 'DefaultErrorHandler', 'max_retries': 10, 'backoff_strategies': [{'type': 'ConstantBackoffStrategy', 'backoff_time_in_seconds': 30}], 'response_filters': [{'predicate': \"{{ response.status_code == 403 and response.get('error', {}).get('type') == 'INVALID_PERMISSIONS_OR_MODEL_NOT_FOUND' }}\", 'action': 'FAIL', 'failure_type': 'config_error', 'error_message': \"{{ 'Personal Access Token does not have required permissions, please add all required permissions to existed one or create new PAT, see docs for more info: https://docs.airbyte.com/integrations/sources/airtable#step-1-set-up-airtable' if config.get('credentials', {}). get('auth_method', '') == 'api_key' else 'Access Token does not have required permissions, please reauthenticate.' }}\", 'type': 'HttpResponseFilter'}, {'http_codes': [403, 422], 'action': 'FAIL', 'failure_type': 'config_error', 'error_message': 'Permission denied or entity is unprocessable.', 'type': 'HttpResponseFilter'}, {'http_codes': [401], 'action': 'FAIL', 'failure_type': 'config_error', 'error_message': 'Invalid credentials were provided. Please re-authenticate to restore access to Airtable.', 'type': 'HttpResponseFilter'}]}, 'path': 'meta/bases/{{ stream_partition.base_id }}/tables', 'http_method': 'GET'}, 'record_selector': {'type': 'RecordSelector', 'extractor': {'type': 'DpathExtractor', 'field_path': ['tables']}, 'pagination_strategy': {'type': 'CursorPagination', 'cursor_value': '{{ response.get(\"offset\", {}) }}', 'stop_condition': '{{ not response.get(\"offset\", {}) }}'}}, 'partition_router': {'type': 'SubstreamPartitionRouter', 'parent_stream_configs': [{'type': 'ParentStreamConfig', 'parent_key': 'id', 'partition_field': 'base_id', 'extra_fields': [['name']], 'stream': {'type': 'DeclarativeStream', 'name': 'bases', 'retriever': {'type': 'SimpleRetriever', 'requester': {'type': 'HttpRequester', 'url_base': 'https://api.airtable.com/v0/', 'authenticator': {'type': 'SelectiveAuthenticator', 'authenticator_selection_path': ['credentials', 'auth_method'], 'authenticators': {'oauth2.0': {'type': 'OAuthAuthenticator', 'client_id': \"{{ config['credentials']['client_id'] }}\", 'client_secret': \"{{ config['credentials']['client_secret'] }}\", 'refresh_token': \"{{ config['credentials']['refresh_token'] }}\", 'grant_type': 'refresh_token', 'refresh_request_headers': {'Authorization': \"Basic {{ [config['credentials']['client_id'], config['credentials']['client_secret']] | join(':') | base64encode }}\", 'Content-Type': 'application/x-www-form-urlencoded'}, 'refresh_token_updater': {'refresh_token_error_status_codes': [400, 401], 'refresh_token_error_key': 'error', 'refresh_token_error_values': ['invalid_grant', 'invalid_client'], 'refresh_token_name': 'refresh_token', 'access_token_config_path': ['credentials', 'access_token'], 'token_expiry_date_config_path': ['credentials', 'token_expiry_date'], 'refresh_token_config_path': ['credentials', 'refresh_token']}, 'token_refresh_endpoint': 'https://airtable.com/oauth2/v1/token'}, 'api_key': {'type': 'BearerAuthenticator', 'api_token': \"{{ config['credentials']['api_key'] }}\"}}}, 'error_handler': {'type': 'DefaultErrorHandler', 'max_retries': 10, 'backoff_strategies': [{'type': 'ConstantBackoffStrategy', 'backoff_time_in_seconds': 30}], 'response_filters': [{'predicate': \"{{ response.status_code == 403 and response.get('error', {}).get('type') == 'INVALID_PERMISSIONS_OR_MODEL_NOT_FOUND' }}\", 'action': 'FAIL', 'failure_type': 'config_error', 'error_message': \"{{ 'Personal Access Token does not have required permissions, please add all required permissions to existed one or create new PAT, see docs for more info: https://docs.airbyte.com/integrations/sources/airtable#step-1-set-up-airtable' if config.get('credentials', {}). get('auth_method', '') == 'api_key' else 'Access Token does not have required permissions, please reauthenticate.' }}\", 'type': 'HttpResponseFilter'}, {'http_codes': [403, 422], 'action': 'FAIL', 'failure_type': 'config_error', 'error_message': 'Permission denied or entity is unprocessable.', 'type': 'HttpResponseFilter'}, {'http_codes': [401], 'action': 'FAIL', 'failure_type': 'config_error', 'error_message': 'Invalid credentials were provided. Please re-authenticate to restore access to Airtable.', 'type': 'HttpResponseFilter'}]}, 'path': 'meta/bases', 'http_method': 'GET'}, 'record_selector': {'type': 'RecordSelector', 'extractor': {'type': 'DpathExtractor', 'field_path': ['bases']}, 'record_filter': {'type': 'RecordFilter', 'condition': '{{ record.get(\"permissionLevel\") }}'}}, 'paginator': {'type': 'DefaultPaginator', 'page_token_option': {'type': 'RequestOption', 'inject_into': 'request_parameter', 'field_name': 'offset'}, 'pagination_strategy': {'type': 'CursorPagination', 'cursor_value': '{{ response.get(\"offset\", {}) }}', 'stop_condition': '{{ not response.get(\"offset\", {}) }}'}}}, 'schema_loader': {'type': 'InlineSchemaLoader', 'schema': {'type': 'object', '$schema': 'http://json-schema.org/schema#', 'additionalProperties': True, 'properties': {'id': {'type': ['string', 'null']}, 'name': {'type': ['string', 'null']}, 'permissionLevel': {'type': ['string', 'null']}}}}}}]}}, 'components_mapping': [{'type': 'ComponentMappingDefinition', 'field_path': ['name'], 'value': \"{{ stream_slice.extra_fields.name.replace(' ', '_').lower().strip() }}/{{ components_values.name.replace(' ', '_').lower().strip() }}/{{ components_values.id }}\"}, {'type': 'ComponentMappingDefinition', 'field_path': ['retriever', 'requester', 'path'], 'value': '{{ stream_slice.base_id }}/{{ components_values.id }}'}, {'type': 'ComponentMappingDefinition', 'field_path': ['schema_loader', 'retriever', 'requester', 'path'], 'value': '{{ stream_slice.base_id }}/{{ components_values.id }}'}, {'type': 'ComponentMappingDefinition', 'field_path': ['transformations', 0, 'fields', 0, 'value'], 'value': '{{ components_values.name }}'}, {'type': 'ComponentMappingDefinition', 'field_path': ['schema_loader', 'retriever', 'requester', 'path'], 'value': 'meta/bases/{{ stream_slice.base_id }}/tables'}, {'type': 'ComponentMappingDefinition', 'field_path': ['schema_loader', 'retriever', 'record_selector', 'record_filter', '$parameters', 'table_id'], 'value': '{{ components_values.id }}'}]} is not valid under any of the given schemas\n\nFailed validating 'anyOf' in schema['properties']['dynamic_streams']['items']['properties']['components_resolver']:\n {'anyOf': [{'$ref': '#/definitions/HttpComponentsResolver'},\n {'$ref': '#/definitions/ConfigComponentsResolver'}],\n 'description': 'Component resolve and populates stream templates with '\n 'components values.',\n 'title': 'Components Resolver'}\n\nOn instance['dynamic_streams'][0]['components_resolver']:\n {'components_mapping': [{'field_path': ['name'],\n 'type': 'ComponentMappingDefinition',\n 'value': '{{ '\n \"stream_slice.extra_fields.name.replace(' \"\n \"', '_').lower().strip() }}/{{ \"\n \"components_values.name.replace(' ', \"\n \"'_').lower().strip() }}/{{ \"\n 'components_values.id }}'},\n {'field_path': ['retriever',\n 'requester',\n 'path'],\n 'type': 'ComponentMappingDefinition',\n 'value': '{{ stream_slice.base_id }}/{{ '\n 'components_values.id }}'},\n {'field_path': ['schema_loader',\n 'retriever',\n 'requester',\n 'path'],\n 'type': 'ComponentMappingDefinition',\n 'value': '{{ stream_slice.base_id }}/{{ '\n 'components_values.id }}'},\n {'field_path': ['transformations',\n 0,\n 'fields',\n 0,\n 'value'],\n 'type': 'ComponentMappingDefinition',\n 'value': '{{ components_values.name }}'},\n {'field_path': ['schema_loader',\n 'retriever',\n 'requester',\n 'path'],\n 'type': 'ComponentMappingDefinition',\n 'value': 'meta/bases/{{ stream_slice.base_id '\n '}}/tables'},\n {'field_path': ['schema_loader',\n 'retriever',\n 'record_selector',\n 'record_filter',\n '$parameters',\n 'table_id'],\n 'type': 'ComponentMappingDefinition',\n 'value': '{{ components_values.id }}'}],\n 'retriever': {'partition_router': {'parent_stream_configs': [{'extra_fields': [['name']],\n 'parent_key': 'id',\n 'partition_field': 'base_id',\n 'stream': {'name': 'bases',\n 'retriever': {'paginator': {'page_token_option': {'field_name': 'offset',\n 'inject_into': 'request_parameter',\n 'type': 'RequestOption'},\n 'pagination_strategy': {'cursor_value': '{{ '\n 'response.get(\"offset\", '\n '{}) '\n '}}',\n 'stop_condition': '{{ '\n 'not '\n 'response.get(\"offset\", '\n '{}) '\n '}}',\n 'type': 'CursorPagination'},\n 'type': 'DefaultPaginator'},\n 'record_selector': {'extractor': {'field_path': ['bases'],\n 'type': 'DpathExtractor'},\n 'record_filter': {'condition': '{{ '\n 'record.get(\"permissionLevel\") '\n '}}',\n 'type': 'RecordFilter'},\n 'type': 'RecordSelector'},\n 'requester': {'authenticator': {'authenticator_selection_path': ['credentials',\n 'auth_method'],\n 'authenticators': {'api_key': {'api_token': '{{ '\n \"config['credentials']['api_key'] \"\n '}}',\n 'type': 'BearerAuthenticator'},\n 'oauth2.0': {'client_id': '{{ '\n \"config['credentials']['client_id'] \"\n '}}',\n 'client_secret': '{{ '\n \"config['credentials']['client_secret'] \"\n '}}',\n 'grant_type': 'refresh_token',\n 'refresh_request_headers': {'Authorization': 'Basic '\n '{{ '\n \"[config['credentials']['client_id'], \"\n \"config['credentials']['client_secret']] \"\n '| '\n \"join(':') \"\n '| '\n 'base64encode '\n '}}',\n 'Content-Type': 'application/x-www-form-urlencoded'},\n 'refresh_token': '{{ '\n \"config['credentials']['refresh_token'] \"\n '}}',\n 'refresh_token_updater': {'access_token_config_path': ['credentials',\n 'access_token'],\n 'refresh_token_config_path': ['credentials',\n 'refresh_token'],\n 'refresh_token_error_key': 'error',\n 'refresh_token_error_status_codes': [400,\n 401],\n 'refresh_token_error_values': ['invalid_grant',\n 'invalid_client'],\n 'refresh_token_name': 'refresh_token',\n 'token_expiry_date_config_path': ['credentials',\n 'token_expiry_date']},\n 'token_refresh_endpoint': 'https://airtable.com/oauth2/v1/token',\n 'type': 'OAuthAuthenticator'}},\n 'type': 'SelectiveAuthenticator'},\n 'error_handler': {'backoff_strategies': [{'backoff_time_in_seconds': 30,\n 'type': 'ConstantBackoffStrategy'}],\n 'max_retries': 10,\n 'response_filters': [{'action': 'FAIL',\n 'error_message': '{{ '\n \"'Personal \"\n 'Access '\n 'Token '\n 'does '\n 'not '\n 'have '\n 'required '\n 'permissions, '\n 'please '\n 'add '\n 'all '\n 'required '\n 'permissions '\n 'to '\n 'existed '\n 'one '\n 'or '\n 'create '\n 'new '\n 'PAT, '\n 'see '\n 'docs '\n 'for '\n 'more '\n 'info: '\n \"https://docs.airbyte.com/integrations/sources/airtable#step-1-set-up-airtable' \"\n 'if '\n \"config.get('credentials', \"\n '{}). '\n \"get('auth_method', \"\n \"'') \"\n '== '\n \"'api_key' \"\n 'else '\n \"'Access \"\n 'Token '\n 'does '\n 'not '\n 'have '\n 'required '\n 'permissions, '\n 'please '\n \"reauthenticate.' \"\n '}}',\n 'failure_type': 'config_error',\n 'predicate': '{{ '\n 'response.status_code '\n '== '\n '403 '\n 'and '\n \"response.get('error', \"\n \"{}).get('type') \"\n '== '\n \"'INVALID_PERMISSIONS_OR_MODEL_NOT_FOUND' \"\n '}}',\n 'type': 'HttpResponseFilter'},\n {'action': 'FAIL',\n 'error_message': 'Permission '\n 'denied '\n 'or '\n 'entity '\n 'is '\n 'unprocessable.',\n 'failure_type': 'config_error',\n 'http_codes': [403,\n 422],\n 'type': 'HttpResponseFilter'},\n {'action': 'FAIL',\n 'error_message': 'Invalid '\n 'credentials '\n 'were '\n 'provided. '\n 'Please '\n 're-authenticate '\n 'to '\n 'restore '\n 'access '\n 'to '\n 'Airtable.',\n 'failure_type': 'config_error',\n 'http_codes': [401],\n 'type': 'HttpResponseFilter'}],\n 'type': 'DefaultErrorHandler'},\n 'http_method': 'GET',\n 'path': 'meta/bases',\n 'type': 'HttpRequester',\n 'url_base': 'https://api.airtable.com/v0/'},\n 'type': 'SimpleRetriever'},\n 'schema_loader': {'schema': {'$schema': 'http://json-schema.org/schema#',\n 'additionalProperties': True,\n 'properties': {'id': {'type': ['string',\n 'null']},\n 'name': {'type': ['string',\n 'null']},\n 'permissionLevel': {'type': ['string',\n 'null']}},\n 'type': 'object'},\n 'type': 'InlineSchemaLoader'},\n 'type': 'DeclarativeStream'},\n 'type': 'ParentStreamConfig'}],\n 'type': 'SubstreamPartitionRouter'},\n 'record_selector': {'extractor': {'field_path': ['tables'],\n 'type': 'DpathExtractor'},\n 'pagination_strategy': {'cursor_value': '{{ '\n 'response.get(\"offset\", '\n '{}) '\n '}}',\n 'stop_condition': '{{ '\n 'not '\n 'response.get(\"offset\", '\n '{}) '\n '}}',\n 'type': 'CursorPagination'},\n 'type': 'RecordSelector'},\n 'requester': {'authenticator': {'authenticator_selection_path': ['credentials',\n 'auth_method'],\n 'authenticators': {'api_key': {'api_token': '{{ '\n \"config['credentials']['api_key'] \"\n '}}',\n 'type': 'BearerAuthenticator'},\n 'oauth2.0': {'client_id': '{{ '\n \"config['credentials']['client_id'] \"\n '}}',\n 'client_secret': '{{ '\n \"config['credentials']['client_secret'] \"\n '}}',\n 'grant_type': 'refresh_token',\n 'refresh_request_headers': {'Authorization': 'Basic '\n '{{ '\n \"[config['credentials']['client_id'], \"\n \"config['credentials']['client_secret']] \"\n '| '\n \"join(':') \"\n '| '\n 'base64encode '\n '}}',\n 'Content-Type': 'application/x-www-form-urlencoded'},\n 'refresh_token': '{{ '\n \"config['credentials']['refresh_token'] \"\n '}}',\n 'refresh_token_updater': {'access_token_config_path': ['credentials',\n 'access_token'],\n 'refresh_token_config_path': ['credentials',\n 'refresh_token'],\n 'refresh_token_error_key': 'error',\n 'refresh_token_error_status_codes': [400,\n 401],\n 'refresh_token_error_values': ['invalid_grant',\n 'invalid_client'],\n 'refresh_token_name': 'refresh_token',\n 'token_expiry_date_config_path': ['credentials',\n 'token_expiry_date']},\n 'token_refresh_endpoint': 'https://airtable.com/oauth2/v1/token',\n 'type': 'OAuthAuthenticator'}},\n 'type': 'SelectiveAuthenticator'},\n 'error_handler': {'backoff_strategies': [{'backoff_time_in_seconds': 30,\n 'type': 'ConstantBackoffStrategy'}],\n 'max_retries': 10,\n 'response_filters': [{'action': 'FAIL',\n 'error_message': '{{ '\n \"'Personal \"\n 'Access '\n 'Token '\n 'does '\n 'not '\n 'have '\n 'required '\n 'permissions, '\n 'please '\n 'add '\n 'all '\n 'required '\n 'permissions '\n 'to '\n 'existed '\n 'one '\n 'or '\n 'create '\n 'new '\n 'PAT, '\n 'see '\n 'docs '\n 'for '\n 'more '\n 'info: '\n \"https://docs.airbyte.com/integrations/sources/airtable#step-1-set-up-airtable' \"\n 'if '\n \"config.get('credentials', \"\n '{}). '\n \"get('auth_method', \"\n \"'') \"\n '== '\n \"'api_key' \"\n 'else '\n \"'Access \"\n 'Token '\n 'does '\n 'not '\n 'have '\n 'required '\n 'permissions, '\n 'please '\n \"reauthenticate.' \"\n '}}',\n 'failure_type': 'config_error',\n 'predicate': '{{ '\n 'response.status_code '\n '== '\n '403 '\n 'and '\n \"response.get('error', \"\n \"{}).get('type') \"\n '== '\n \"'INVALID_PERMISSIONS_OR_MODEL_NOT_FOUND' \"\n '}}',\n 'type': 'HttpResponseFilter'},\n {'action': 'FAIL',\n 'error_message': 'Permission '\n 'denied '\n 'or '\n 'entity '\n 'is '\n 'unprocessable.',\n 'failure_type': 'config_error',\n 'http_codes': [403,\n 422],\n 'type': 'HttpResponseFilter'},\n {'action': 'FAIL',\n 'error_message': 'Invalid '\n 'credentials '\n 'were '\n 'provided. '\n 'Please '\n 're-authenticate '\n 'to '\n 'restore '\n 'access '\n 'to '\n 'Airtable.',\n 'failure_type': 'config_error',\n 'http_codes': [401],\n 'type': 'HttpResponseFilter'}],\n 'type': 'DefaultErrorHandler'},\n 'http_method': 'GET',\n 'path': 'meta/bases/{{ '\n 'stream_partition.base_id '\n '}}/tables',\n 'type': 'HttpRequester',\n 'url_base': 'https://api.airtable.com/v0/'},\n 'type': 'SimpleRetriever'},\n 'type': 'HttpComponentsResolver'}\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File \"/Users/ajsteers/.local/share/uv/tools/airbyte-source-airtable/lib/python3.12/site-packages/source_airtable/run.py\", line 25, in _get_source\n return SourceAirtable(\n ^^^^^^^^^^^^^^^\n File \"/Users/ajsteers/.local/share/uv/tools/airbyte-source-airtable/lib/python3.12/site-packages/source_airtable/source.py\", line 13, in __init__\n super().__init__(catalog=catalog, config=config, state=state, **{\"path_to_yaml\": \"manifest.yaml\"})\n File \"/Users/ajsteers/.local/share/uv/tools/airbyte-source-airtable/lib/python3.12/site-packages/airbyte_cdk/sources/declarative/yaml_declarative_source.py\", line 34, in __init__\n super().__init__(\n File \"/Users/ajsteers/.local/share/uv/tools/airbyte-source-airtable/lib/python3.12/site-packages/airbyte_cdk/sources/declarative/concurrent_declarative_source.py\", line 93, in __init__\n super().__init__(\n File \"/Users/ajsteers/.local/share/uv/tools/airbyte-source-airtable/lib/python3.12/site-packages/airbyte_cdk/sources/declarative/manifest_declarative_source.py\", line 138, in __init__\n self._validate_source()\n File \"/Users/ajsteers/.local/share/uv/tools/airbyte-source-airtable/lib/python3.12/site-packages/airbyte_cdk/sources/declarative/manifest_declarative_source.py\", line 396, in _validate_source\n raise ValidationError(\njsonschema.exceptions.ValidationError: Validation against json schema defined in declarative_component_schema.yaml schema failed\n"}}}`
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.