airbytehq / airbytehq/airbyte

Streams without a status message

未关闭
#72,367 10 条评论 1 个 reaction 已指派 0 人 在 GitHub 查看
area/connectors autoteam community connectors/destination/bigquery needs-triage team/extensibility team/use type/bug
主要语言
Python
星标
22.1k
派生
5.3k
PR 合并指标
PR 指标待抓取

描述

### Connector Name

Big Query

### Connector Version

BigQuery v3.0.16

### What step the error happened?

During the sync

### Relevant information

# [source-declarative-manifest] Streams do not emit terminal stream status messages

## Connector Name

`source-declarative-manifest`

## Connector Version

`7.6.5` (used in Airbyte Cloud)

## Destination Connector

`destination-bigquery:3.0.16` (CDK v3)

## What step the error happened?

During the sync operation (replication)

## Description

When using `source-declarative-manifest` with a declarative YAML source that defines multiple streams (or even a single stream), the connector does not emit terminal stream status messages that are required by destinations using CDK v3 (like `destination-bigquery:3.0.16`).

The sync fails with the error:

```
io.airbyte.cdk.TransientErrorException: Input was fully read, but some streams did not receive a terminal stream status message.
If the destination did not encounter other errors, this likely indicates an error in the source or platform.
Streams without a status message: [applications]
```

## Expected Behavior

The `source-declarative-manifest` should automatically emit stream status messages (specifically `STREAM_STATUS` messages) when a stream finishes reading all records, similar to how native Python connectors do.

## Actual Behavior

The connector reads all data successfully (authentication works, pagination works, records are read), but does not emit the terminal stream status message that destinations with CDK v3 require. This causes the destination to fail even though all data was successfully read.

## Relevant Information

### YAML Configuration

The declarative source YAML defines streams using `DeclarativeStream` with:

- OAuth2 authentication (Basic Auth + client_credentials)
- Pagination using `DefaultPaginator` with `PageIncrement`
- Simple record extraction from API responses

### Error Details

**Error during CHECK operation:**

- ✅ CHECK passes successfully
- ✅ Connection test works
- ✅ Discover works

**Error during SYNC operation:**

- ✅ Authentication works
- ✅ Data is read from API
- ✅ Records are sent to destination
- ❌ **Stream status messages are NOT emitted**
- ❌ Destination fails waiting for stream status

### Logs

```
2026-01-27 00:26:42 error io.airbyte.cdk.TransientErrorException: Input was fully read, but some streams did not receive a terminal stream status message.
If the destination did not encounter other errors, this likely indicates an error in the source or platform.
Streams without a status message: [applications]
at io.airbyte.cdk.load.state.SyncManager.markInputConsumed(SyncManager.kt:116)
at io.airbyte.cdk.load.state.PipelineEventBookkeepingRouter.close(PipelineEventBookkeepingRouter.kt:300)
at io.airbyte.cdk.load.task.internal.InputConsumerTask.execute(InputConsumerTask.kt:117)
```

### Workaround Attempts

1. ✅ Tried with single stream (removed second stream) - **Still fails**
2. ✅ Verified authentication is working correctly
3. ✅ Verified pagination is working correctly
4. ✅ Verified data is being read successfully
5. ❌ No YAML configuration option exists to force stream status emission

### Related Issues

This appears to be the same issue affecting other connectors:

- [#61419](https://github.com/airbytehq/airbyte/issues/61419) - Similar error with `source-clickhouse`

## Impact

- **Severity**: High - Blocks all syncs using `source-declarative-manifest` with destinations on CDK v3
- **Affected Users**: Anyone using declarative YAML sources with modern destinations (BigQuery 3.x, Snowflake 3.x, etc.)
- **Workaround**: None available in Airbyte Cloud (cannot downgrade destination versions)

## Environment

- **Platform**: Airbyte Cloud
- **Source**: `source-declarative-manifest:7.6.5`
- **Destination**: `destination-bigquery:3.0.16`
- **Data Volume**: ~170 records (small dataset, not a scale issue)

## Additional Context

The issue is that `source-declarative-manifest` does not implement the emission of stream status messages that are required by the new CDK v3 destinations. This is a code-level limitation, not a YAML configuration issue.

**Suggested Fix:**
The `source-declarative-manifest` connector should be updated to emit `STREAM_STATUS` messages when each stream completes, similar to how native Python connectors handle this.

### Relevant log output

```shell

```

### Contribute

- [ ] Yes, I want to contribute

---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/11046

贡献指南

打开贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。