Streams without a status message
- 主要语言
- Python
- 星标
- 22.1k
- 派生
- 5.3k
- PR 合并指标
- PR 指标待抓取
描述
### Connector Name
Big Query
### Connector Version
BigQuery v3.0.16
### What step the error happened?
During the sync
### Relevant information
# [source-declarative-manifest] Streams do not emit terminal stream status messages
## Connector Name
`source-declarative-manifest`
## Connector Version
`7.6.5` (used in Airbyte Cloud)
## Destination Connector
`destination-bigquery:3.0.16` (CDK v3)
## What step the error happened?
During the sync operation (replication)
## Description
When using `source-declarative-manifest` with a declarative YAML source that defines multiple streams (or even a single stream), the connector does not emit terminal stream status messages that are required by destinations using CDK v3 (like `destination-bigquery:3.0.16`).
The sync fails with the error:
```
io.airbyte.cdk.TransientErrorException: Input was fully read, but some streams did not receive a terminal stream status message.
If the destination did not encounter other errors, this likely indicates an error in the source or platform.
Streams without a status message: [applications]
```
## Expected Behavior
The `source-declarative-manifest` should automatically emit stream status messages (specifically `STREAM_STATUS` messages) when a stream finishes reading all records, similar to how native Python connectors do.
## Actual Behavior
The connector reads all data successfully (authentication works, pagination works, records are read), but does not emit the terminal stream status message that destinations with CDK v3 require. This causes the destination to fail even though all data was successfully read.
## Relevant Information
### YAML Configuration
The declarative source YAML defines streams using `DeclarativeStream` with:
- OAuth2 authentication (Basic Auth + client_credentials)
- Pagination using `DefaultPaginator` with `PageIncrement`
- Simple record extraction from API responses
### Error Details
**Error during CHECK operation:**
- ✅ CHECK passes successfully
- ✅ Connection test works
- ✅ Discover works
**Error during SYNC operation:**
- ✅ Authentication works
- ✅ Data is read from API
- ✅ Records are sent to destination
- ❌ **Stream status messages are NOT emitted**
- ❌ Destination fails waiting for stream status
### Logs
```
2026-01-27 00:26:42 error io.airbyte.cdk.TransientErrorException: Input was fully read, but some streams did not receive a terminal stream status message.
If the destination did not encounter other errors, this likely indicates an error in the source or platform.
Streams without a status message: [applications]
at io.airbyte.cdk.load.state.SyncManager.markInputConsumed(SyncManager.kt:116)
at io.airbyte.cdk.load.state.PipelineEventBookkeepingRouter.close(PipelineEventBookkeepingRouter.kt:300)
at io.airbyte.cdk.load.task.internal.InputConsumerTask.execute(InputConsumerTask.kt:117)
```
### Workaround Attempts
1. ✅ Tried with single stream (removed second stream) - **Still fails**
2. ✅ Verified authentication is working correctly
3. ✅ Verified pagination is working correctly
4. ✅ Verified data is being read successfully
5. ❌ No YAML configuration option exists to force stream status emission
### Related Issues
This appears to be the same issue affecting other connectors:
- [#61419](https://github.com/airbytehq/airbyte/issues/61419) - Similar error with `source-clickhouse`
## Impact
- **Severity**: High - Blocks all syncs using `source-declarative-manifest` with destinations on CDK v3
- **Affected Users**: Anyone using declarative YAML sources with modern destinations (BigQuery 3.x, Snowflake 3.x, etc.)
- **Workaround**: None available in Airbyte Cloud (cannot downgrade destination versions)
## Environment
- **Platform**: Airbyte Cloud
- **Source**: `source-declarative-manifest:7.6.5`
- **Destination**: `destination-bigquery:3.0.16`
- **Data Volume**: ~170 records (small dataset, not a scale issue)
## Additional Context
The issue is that `source-declarative-manifest` does not implement the emission of stream status messages that are required by the new CDK v3 destinations. This is a code-level limitation, not a YAML configuration issue.
**Suggested Fix:**
The `source-declarative-manifest` connector should be updated to emit `STREAM_STATUS` messages when each stream completes, similar to how native Python connectors handle this.
### Relevant log output
```shell
```
### Contribute
- [ ] Yes, I want to contribute
---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/11046
贡献指南
评估
这个 Issue 还没有评估数据。