airbytehq / airbytehq/airbyte

[Feature Request] Allow skipping polling stage in AsyncRetriever when creation response provides download URL

Offen
#70,329 1 Kommentar 1 Reaktion 3 zugewiesene Personen Beansprucht von @syedkhadeer-cmd Auf GitHub ansehen
team/triage
Vorherrschende Sprache
Python
Sterne
22.1k
Forks
5.3k
PR-Merge-Kennzahlen
PR-Kennzahlen ausstehend

Beschreibung

## Summary

When configuring an asynchronous job in the Connector Builder, some APIs directly provide a download URL in the creation response, making the polling stage unnecessary. Currently, the `polling_requester` is a required field in the `AsyncRetriever` schema, forcing users to configure a polling endpoint even when it's not needed.

## Problem

A customer reported struggling to configure an async job where the API directly provides a download URL during the creation stage. The current workaround involves:
1. Using the creation response variable in the Download tab
2. Adding a dummy URL (e.g., a fake GitHub status page URL) in the Polling tab to satisfy the required field

While this workaround functions, it's not an ideal solution and creates confusion for users.

## Technical Details

Show/Hide Technical Analysis

The `polling_requester` is defined as a required field in the `AsyncRetriever` schema:

```python
# declarative_component_schema.py:2353-2356
polling_requester: Union[CustomRequester, HttpRequester] = Field(
...,
description="Requester component that describes how to prepare HTTP requests to send to the source API to fetch the status of the running async job.",
)
```

The `AsyncHttpJobRepository` class in `http_job_repository.py` always expects a polling response to extract download targets:

```python
# http_job_repository.py:332-334
def _get_download_targets(self, job: AsyncJob) -> Iterable[str]:
if not self.download_target_requester:
url_response = self._polling_job_response_by_id[job.api_job_id()]
```

## Proposed Solution

Make the `polling_requester` field optional in the `AsyncRetriever` schema and update the `AsyncHttpJobRepository` to handle the case where:
1. No polling is configured
2. The download URL is extracted directly from the creation response
3. The job is immediately marked as "completed" after creation

This would allow users to configure the `download_target_extractor` to use `{{ creation_response.download_url }}` (or similar) without needing a dummy polling endpoint.

## Related Issues

- https://github.com/airbytehq/airbyte/issues/55843 - [Roadmap] Support Async data format in Connector Builder (closed)
- https://github.com/airbytehq/airbyte/issues/65127 - Async Polling Streams timeout issues (open)

## Requested by

syed.khadeer@airbyte.io (@Airbyte-Support) on behalf of a customer

## Investigation Link

https://app.devin.ai/sessions/7e6f6d6a45b44166bdbaf062154b7771

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.