airbytehq / airbytehq/airbyte

[Feature Request] Allow skipping polling stage in AsyncRetriever when creation response provides download URL

未關閉
#70,329 1 則留言 1 個 reaction 已指派 3 人 已被 @syedkhadeer-cmd 認領 在 GitHub 檢視
team/triage
主要語言
Python
星號
22.1k
分支
5.4k
平均合併
5 小時
30 天內合併 PR
671

描述

## Summary

When configuring an asynchronous job in the Connector Builder, some APIs directly provide a download URL in the creation response, making the polling stage unnecessary. Currently, the `polling_requester` is a required field in the `AsyncRetriever` schema, forcing users to configure a polling endpoint even when it's not needed.

## Problem

A customer reported struggling to configure an async job where the API directly provides a download URL during the creation stage. The current workaround involves:
1. Using the creation response variable in the Download tab
2. Adding a dummy URL (e.g., a fake GitHub status page URL) in the Polling tab to satisfy the required field

While this workaround functions, it's not an ideal solution and creates confusion for users.

## Technical Details

Show/Hide Technical Analysis

The `polling_requester` is defined as a required field in the `AsyncRetriever` schema:

```python
# declarative_component_schema.py:2353-2356
polling_requester: Union[CustomRequester, HttpRequester] = Field(
...,
description="Requester component that describes how to prepare HTTP requests to send to the source API to fetch the status of the running async job.",
)
```

The `AsyncHttpJobRepository` class in `http_job_repository.py` always expects a polling response to extract download targets:

```python
# http_job_repository.py:332-334
def _get_download_targets(self, job: AsyncJob) -> Iterable[str]:
if not self.download_target_requester:
url_response = self._polling_job_response_by_id[job.api_job_id()]
```

## Proposed Solution

Make the `polling_requester` field optional in the `AsyncRetriever` schema and update the `AsyncHttpJobRepository` to handle the case where:
1. No polling is configured
2. The download URL is extracted directly from the creation response
3. The job is immediately marked as "completed" after creation

This would allow users to configure the `download_target_extractor` to use `{{ creation_response.download_url }}` (or similar) without needing a dummy polling endpoint.

## Related Issues

- https://github.com/airbytehq/airbyte/issues/55843 - [Roadmap] Support Async data format in Connector Builder (closed)
- https://github.com/airbytehq/airbyte/issues/65127 - Async Polling Streams timeout issues (open)

## Requested by

syed.khadeer@airbyte.io (@Airbyte-Support) on behalf of a customer

## Investigation Link

https://app.devin.ai/sessions/7e6f6d6a45b44166bdbaf062154b7771

貢獻指南

開啟貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。