[AsyncRetriever] - Partition router is not working when using the AsyncRetriever
- Ngôn ngữ chính
- Python
- Star
- 22.1k
- Fork
- 5.3k
- Chỉ số merge pull request
- Chỉ số pull request đang chờ
Mô tả
### Topic
The request parameters defined in the partition router are not being sent in creation_requester
### Relevant information
I've started using the AsyncRetriever to create a low code connector in order to integrate a data source.
I have 2 inputs in my configuration file and I'm also using the incremental sync + start_time_option to insert a date as parameter, none of the these variables are being sent to my first request.
Although, when I hard code this parameters in my base request the connector works, the ideal scenario is to use the partitions.
This is my retriever code:
```yaml
retriever:
type: AsyncRetriever
status_mapping:
failed:
- '201'
running:
- '202'
timeout:
- '203'
completed:
- '200'
urls_extractor:
type: DpathExtractor
field_path:
- url
record_selector:
type: RecordSelector
extractor:
type: DpathExtractor
field_path:
- data
status_extractor:
type: DpathExtractor
field_path:
- status
creation_requester:
$ref: "#/definitions/base_requester"
path: userAdRevenueReport
http_method: GET
polling_requester:
$ref: "#/definitions/base_requester"
path: userAdRevenueReport
http_method: GET
download_requester:
type: HttpRequester
path: "{{stream_slice['url']}}"
url_base: ""
http_method: GET
partition_router:
- type: ListPartitionRouter
values: '{{ config[''application''] }}'
cursor_field: application
request_option:
type: RequestOption
field_name: application
inject_into: request_parameter
- type: ListPartitionRouter
values: '{{ config[''platform''] }}'
cursor_field: platform
request_option:
type: RequestOption
field_name: platform
inject_into: request_parameter
incremental_sync:
type: DatetimeBasedCursor
cursor_field: date_reference
lookback_window: P{{config['lookback_window__days_']}}D
cursor_datetime_formats:
- '%Y-%m-%d'
datetime_format: '%Y-%m-%d'
start_datetime:
type: MinMaxDatetime
datetime: '{{ config[''start_date''] }}'
datetime_format: '%Y-%m-%dT%H:%M:%SZ'
start_time_option:
type: RequestOption
field_name: date
inject_into: request_parameter
end_datetime:
type: MinMaxDatetime
datetime: '{{ day_delta(-1, format=''%Y-%m-%d'') }}'
datetime_format: '%Y-%m-%d'
step: P1D
cursor_granularity: P1D
```
Hướng dẫn đóng góp
Đánh giá
Issue này chưa được đánh giá.