airbytehq / airbytehq/airbyte

[AsyncRetriever] - Partition router is not working when using the AsyncRetriever

未關閉
#51,533 3 則留言 1 個 reaction 已指派 0 人 在 GitHub 檢視
area/connector-builder community team/marketplace team/use
主要語言
Python
星號
22.1k
分支
5.4k
平均合併
5 小時
30 天內合併 PR
671

描述

### Topic

The request parameters defined in the partition router are not being sent in creation_requester

### Relevant information

I've started using the AsyncRetriever to create a low code connector in order to integrate a data source.

I have 2 inputs in my configuration file and I'm also using the incremental sync + start_time_option to insert a date as parameter, none of the these variables are being sent to my first request.

Although, when I hard code this parameters in my base request the connector works, the ideal scenario is to use the partitions.

This is my retriever code:

```yaml
retriever:
type: AsyncRetriever
status_mapping:
failed:
- '201'
running:
- '202'
timeout:
- '203'
completed:
- '200'
urls_extractor:
type: DpathExtractor
field_path:
- url
record_selector:
type: RecordSelector
extractor:
type: DpathExtractor
field_path:
- data
status_extractor:
type: DpathExtractor
field_path:
- status
creation_requester:
$ref: "#/definitions/base_requester"
path: userAdRevenueReport
http_method: GET
polling_requester:
$ref: "#/definitions/base_requester"
path: userAdRevenueReport
http_method: GET
download_requester:
type: HttpRequester
path: "{{stream_slice['url']}}"
url_base: ""
http_method: GET
partition_router:
- type: ListPartitionRouter
values: '{{ config[''application''] }}'
cursor_field: application
request_option:
type: RequestOption
field_name: application
inject_into: request_parameter
- type: ListPartitionRouter
values: '{{ config[''platform''] }}'
cursor_field: platform
request_option:
type: RequestOption
field_name: platform
inject_into: request_parameter

incremental_sync:
type: DatetimeBasedCursor
cursor_field: date_reference
lookback_window: P{{config['lookback_window__days_']}}D
cursor_datetime_formats:
- '%Y-%m-%d'
datetime_format: '%Y-%m-%d'
start_datetime:
type: MinMaxDatetime
datetime: '{{ config[''start_date''] }}'
datetime_format: '%Y-%m-%dT%H:%M:%SZ'
start_time_option:
type: RequestOption
field_name: date
inject_into: request_parameter
end_datetime:
type: MinMaxDatetime
datetime: '{{ day_delta(-1, format=''%Y-%m-%d'') }}'
datetime_format: '%Y-%m-%d'
step: P1D
cursor_granularity: P1D
```

貢獻指南

開啟貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。