airbytehq / airbytehq/airbyte

[AsyncRetriever] - Partition router is not working when using the AsyncRetriever

Đang mở
#51,533 3 bình luận 1 reaction 0 người được giao Xem trên GitHub
area/connector-builder community team/marketplace team/use
Ngôn ngữ chính
Python
Star
22.1k
Fork
5.3k
Chỉ số merge pull request
Chỉ số pull request đang chờ

Mô tả

### Topic

The request parameters defined in the partition router are not being sent in creation_requester

### Relevant information

I've started using the AsyncRetriever to create a low code connector in order to integrate a data source.

I have 2 inputs in my configuration file and I'm also using the incremental sync + start_time_option to insert a date as parameter, none of the these variables are being sent to my first request.

Although, when I hard code this parameters in my base request the connector works, the ideal scenario is to use the partitions.

This is my retriever code:

```yaml
retriever:
type: AsyncRetriever
status_mapping:
failed:
- '201'
running:
- '202'
timeout:
- '203'
completed:
- '200'
urls_extractor:
type: DpathExtractor
field_path:
- url
record_selector:
type: RecordSelector
extractor:
type: DpathExtractor
field_path:
- data
status_extractor:
type: DpathExtractor
field_path:
- status
creation_requester:
$ref: "#/definitions/base_requester"
path: userAdRevenueReport
http_method: GET
polling_requester:
$ref: "#/definitions/base_requester"
path: userAdRevenueReport
http_method: GET
download_requester:
type: HttpRequester
path: "{{stream_slice['url']}}"
url_base: ""
http_method: GET
partition_router:
- type: ListPartitionRouter
values: '{{ config[''application''] }}'
cursor_field: application
request_option:
type: RequestOption
field_name: application
inject_into: request_parameter
- type: ListPartitionRouter
values: '{{ config[''platform''] }}'
cursor_field: platform
request_option:
type: RequestOption
field_name: platform
inject_into: request_parameter

incremental_sync:
type: DatetimeBasedCursor
cursor_field: date_reference
lookback_window: P{{config['lookback_window__days_']}}D
cursor_datetime_formats:
- '%Y-%m-%d'
datetime_format: '%Y-%m-%d'
start_datetime:
type: MinMaxDatetime
datetime: '{{ config[''start_date''] }}'
datetime_format: '%Y-%m-%dT%H:%M:%SZ'
start_time_option:
type: RequestOption
field_name: date
inject_into: request_parameter
end_datetime:
type: MinMaxDatetime
datetime: '{{ day_delta(-1, format=''%Y-%m-%d'') }}'
datetime_format: '%Y-%m-%d'
step: P1D
cursor_granularity: P1D
```

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.