airbytehq / airbytehq/PyAirbyte

[source-slack] IncrementalMessageStream for threads channel never stops syncing

Đang mở
#430 0 bình luận 0 reaction 0 người được giao Xem trên GitHub
Ngôn ngữ chính
Python
Star
344
Fork
77
Merge trung bình
1 ngày 11 giờ
Pull request đã merge (30 ngày)
35

Mô tả

### Connector Name

source-slack

### Connector Version

1.3.1

### What step the error happened?

During the sync

### Relevant information

Threads channel syncing never stops. It seems to cycle through all the channels, updating the end timestamp each loop (as some time passed while the data was queried), and does that over and over.
Other channels work fine.

Using PyAirbyte:

```python
import airbyte as ab

source = ab.get_source(
"source-slack",
config={
"api_token": "",
"start_date": "2024-01-01T00:00:00Z",
"lookback_window": 1,
"join_channels": False,
},
install_if_missing=True,
)
source.check()
source.select_streams(["channels", "channel_members", "channel_messages", "threads"])

result = source.read()

for name, records in result.streams.items():
print(f"Stream {name}: {len(list(records))} records")
for record in records:
print(record.get("text"))

```

### Relevant log output

```shell
2024-10-22 15:40:57 - INFO - Check succeeded
2024-10-22 15:41:02 - INFO - Starting syncing SourceSlack
2024-10-22 15:41:02 - INFO - Marking stream channels as STARTED
2024-10-22 15:41:02 - INFO - Setting state of SourceSlack stream to {'__ab_no_cursor_state_message': True}
2024-10-22 15:41:02 - INFO - Syncing stream: channels
2024-10-22 15:41:02 - INFO - Marking stream channels as RUNNING
2024-10-22 15:41:02 - INFO - Read 4 records from channels stream
2024-10-22 15:41:02 - INFO - Marking stream channels as STOPPED
2024-10-22 15:41:02 - INFO - Finished syncing channels
2024-10-22 15:41:02 - INFO - SourceSlack runtimes:
Syncing stream channels 0:00:00.365530
2024-10-22 15:41:02 - INFO - Marking stream channel_members as STARTED
2024-10-22 15:41:02 - INFO - Setting state of SourceSlack stream to {'__ab_no_cursor_state_message': True}
2024-10-22 15:41:02 - INFO - Syncing stream: channel_members
2024-10-22 15:41:02 - INFO - Marking stream channel_members as RUNNING
2024-10-22 15:41:03 - INFO - Read 18 records from channel_members stream
2024-10-22 15:41:03 - INFO - Marking stream channel_members as STOPPED
2024-10-22 15:41:03 - INFO - Finished syncing channel_members
2024-10-22 15:41:03 - INFO - SourceSlack runtimes:
Syncing stream channel_members 0:00:00.985201
Syncing stream channels 0:00:00.365530
2024-10-22 15:41:03 - INFO - Marking stream channel_messages as STARTED
2024-10-22 15:41:03 - INFO - Setting state of SourceSlack stream to {'states': [{'partition': {'channel': 'C07Q9KX2U6T', 'parent_slice': {}}, 'cursor': {'float_ts': 1729324525.094199}}, {'partition': {'channel': 'C07QHGM9B1C', 'parent_slice': {}}, 'cursor': {'float_ts': 1729324525.118059}}, {'partition': {'channel': 'C07RCUS2MGQ', 'parent_slice': {}}, 'cursor': {'float_ts': 1729324525.155319}}, {'partition': {'channel': 'C07RFRXUQN5', 'parent_slice': {}}, 'cursor': {'float_ts': 1729283939.911469}}]}
2024-10-22 15:41:03 - INFO - Syncing stream: channel_messages
2024-10-22 15:41:03 - INFO - Marking stream channel_messages as RUNNING
2024-10-22 15:41:04 - INFO - Read 6 records from channel_messages stream
2024-10-22 15:41:04 - INFO - Marking stream channel_messages as STOPPED
2024-10-22 15:41:04 - INFO - Finished syncing channel_messages
2024-10-22 15:41:04 - INFO - SourceSlack runtimes:
Syncing stream channel_members 0:00:00.985201
Syncing stream channel_messages 0:00:01.123485
Syncing stream channels 0:00:00.365530
2024-10-22 15:41:04 - INFO - Marking stream threads as STARTED
2024-10-22 15:41:04 - INFO - Setting state of SourceSlack stream to {}
2024-10-22 15:41:04 - INFO - Syncing stream: threads
2024-10-22 15:41:04 - INFO - Reading the channel: `test-slack-ingestion`
2024-10-22 15:41:04 - INFO - Syncing replies {'channel': 'C07Q9KX2U6T', 'oldest': 1704067200.0, 'latest': 1712707200.0}
2024-10-22 15:41:04 - INFO - Syncing replies {'channel': 'C07Q9KX2U6T', 'oldest': 1712707200.0, 'latest': 1721347200.0}
2024-10-22 15:41:05 - INFO - Syncing replies {'channel': 'C07Q9KX2U6T', 'oldest': 1721347200.0, 'latest': 1729600864.626089}
2024-10-22 15:41:05 - INFO - Marking stream threads as RUNNING
2024-10-22 15:41:07 - INFO - Reading the channel: `social`
2024-10-22 15:41:07 - INFO - Syncing replies {'channel': 'C07QHGM9B1C', 'oldest': 1704067200.0, 'latest': 1712707200.0}
2024-10-22 15:41:07 - INFO - Syncing replies {'channel': 'C07QHGM9B1C', 'oldest': 1712707200.0, 'latest': 1721347200.0}
2024-10-22 15:41:07 - INFO - Syncing replies {'channel': 'C07QHGM9B1C', 'oldest': 1721347200.0, 'latest': 1729600867.384261}
2024-10-22 15:41:08 - INFO - Reading the channel: `all-new-workspace`
2024-10-22 15:41:08 - INFO - Syncing replies {'channel': 'C07RCUS2MGQ', 'oldest': 1704067200.0, 'latest': 1712707200.0}
2024-10-22 15:41:09 - INFO - Syncing replies {'channel': 'C07RCUS2MGQ', 'oldest': 1712707200.0, 'latest': 1721347200.0}
2024-10-22 15:41:09 - INFO - Syncing replies {'channel': 'C07RCUS2MGQ', 'oldest': 1721347200.0, 'latest': 1729600868.85753}
2024-10-22 15:41:10 - INFO - Reading the channel: `support`
2024-10-22 15:41:10 - INFO - Syncing replies {'channel': 'C07RFRXUQN5', 'oldest': 1704067200.0, 'latest': 1712707200.0}
2024-10-22 15:41:10 - INFO - Syncing replies {'channel': 'C07RFRXUQN5', 'oldest': 1712707200.0, 'latest': 1721347200.0}
2024-10-22 15:41:10 - INFO - Syncing replies {'channel': 'C07RFRXUQN5', 'oldest': 1721347200.0, 'latest': 1729600870.328075}
2024-10-22 15:41:12 - INFO - Reading the channel: `test-slack-ingestion`
2024-10-22 15:41:12 - INFO - Syncing replies {'channel': 'C07Q9KX2U6T', 'oldest': 1704067200.0, 'latest': 1712707200.0}
2024-10-22 15:41:12 - INFO - Syncing replies {'channel': 'C07Q9KX2U6T', 'oldest': 1712707200.0, 'latest': 1721347200.0}
2024-10-22 15:41:12 - INFO - Syncing replies {'channel': 'C07Q9KX2U6T', 'oldest': 1721347200.0, 'latest': 1729600872.45988}
2024-10-22 15:41:14 - INFO - Reading the channel: `social`
2024-10-22 15:41:14 - INFO - Syncing replies {'channel': 'C07QHGM9B1C', 'oldest': 1704067200.0, 'latest': 1712707200.0}
2024-10-22 15:41:14 - INFO - Syncing replies {'channel': 'C07QHGM9B1C', 'oldest': 1712707200.0, 'latest': 1721347200.0}
2024-10-22 15:41:14 - INFO - Syncing replies {'channel': 'C07QHGM9B1C', 'oldest': 1721347200.0, 'latest': 1729600874.548036}
2024-10-22 15:41:15 - INFO - Reading the channel: `all-new-workspace`
2024-10-22 15:41:15 - INFO - Syncing replies {'channel': 'C07RCUS2MGQ', 'oldest': 1704067200.0, 'latest': 1712707200.0}
2024-10-22 15:41:15 - INFO - Syncing replies {'channel': 'C07RCUS2MGQ', 'oldest': 1712707200.0, 'latest': 1721347200.0}
2024-10-22 15:41:15 - INFO - Syncing replies {'channel': 'C07RCUS2MGQ', 'oldest': 1721347200.0, 'latest': 1729600875.567569}
2024-10-22 15:41:16 - INFO - Reading the channel: `support`
2024-10-22 15:41:16 - INFO - Syncing replies {'channel': 'C07RFRXUQN5', 'oldest': 1704067200.0, 'latest': 1712707200.0}
2024-10-22 15:41:16 - INFO - Syncing replies {'channel': 'C07RFRXUQN5', 'oldest': 1712707200.0, 'latest': 1721347200.0}
2024-10-22 15:41:16 - INFO - Syncing replies {'channel': 'C07RFRXUQN5', 'oldest': 1721347200.0, 'latest': 1729600876.609494}
2024-10-22 15:41:18 - INFO - Reading the channel: `test-slack-ingestion`
2024-10-22 15:41:18 - INFO - Syncing replies {'channel': 'C07Q9KX2U6T', 'oldest': 1704067200.0, 'latest': 1712707200.0}
2024-10-22 15:41:18 - INFO - Syncing replies {'channel': 'C07Q9KX2U6T', 'oldest': 1712707200.0, 'latest': 1721347200.0}
2024-10-22 15:41:18 - INFO - Syncing replies {'channel': 'C07Q9KX2U6T', 'oldest': 1721347200.0, 'latest': 1729600878.290911}
2024-10-22 15:41:20 - INFO - Reading the channel: `social`
2024-10-22 15:41:20 - INFO - Syncing replies {'channel': 'C07QHGM9B1C', 'oldest': 1704067200.0, 'latest': 1712707200.0}
2024-10-22 15:41:20 - INFO - Syncing replies {'channel': 'C07QHGM9B1C', 'oldest': 1712707200.0, 'latest': 1721347200.0}
2024-10-22 15:41:20 - INFO - Syncing replies {'channel': 'C07QHGM9B1C', 'oldest': 1721347200.0, 'latest': 1729600880.449153}
2024-10-22 15:41:21 - INFO - Reading the channel: `all-new-workspace`
2024-10-22 15:41:21 - INFO - Syncing replies {'channel': 'C07RCUS2MGQ', 'oldest': 1704067200.0, 'latest': 1712707200.0}
2024-10-22 15:41:21 - INFO - Syncing replies {'channel': 'C07RCUS2MGQ', 'oldest': 1712707200.0, 'latest': 1721347200.0}
2024-10-22 15:41:21 - INFO - Syncing replies {'channel': 'C07RCUS2MGQ', 'oldest': 1721347200.0, 'latest': 1729600881.485752}
2024-10-22 15:41:22 - INFO - Reading the channel: `support`
2024-10-22 15:41:22 - INFO - Syncing replies {'channel': 'C07RFRXUQN5', 'oldest': 1704067200.0, 'latest': 1712707200.0}
2024-10-22 15:41:22 - INFO - Syncing replies {'channel': 'C07RFRXUQN5', 'oldest': 1712707200.0, 'latest': 1721347200.0}
2024-10-22 15:41:22 - INFO - Syncing replies {'channel': 'C07RFRXUQN5', 'oldest': 1721347200.0, 'latest': 1729600882.548746}
2024-10-22 15:41:24 - INFO - Reading the channel: `test-slack-ingestion`
2024-10-22 15:41:24 - INFO - Syncing replies {'channel': 'C07Q9KX2U6T', 'oldest': 1704067200.0, 'latest': 1712707200.0}
2024-10-22 15:41:24 - INFO - Syncing replies {'channel': 'C07Q9KX2U6T', 'oldest': 1712707200.0, 'latest': 1721347200.0}
2024-10-22 15:41:24 - INFO - Syncing replies {'channel': 'C07Q9KX2U6T', 'oldest': 1721347200.0, 'latest': 1729600884.416193}
2024-10-22 15:41:27 - INFO - Reading the channel: `social`
2024-10-22 15:41:27 - INFO - Syncing replies {'channel': 'C07QHGM9B1C', 'oldest': 1704067200.0, 'latest': 1712707200.0}
2024-10-22 15:41:27 - INFO - Syncing replies {'channel': 'C07QHGM9B1C', 'oldest': 1712707200.0, 'latest': 1721347200.0}
2024-10-22 15:41:27 - INFO - Syncing replies {'channel': 'C07QHGM9B1C', 'oldest': 1721347200.0, 'latest': 1729600887.079985}
```

### Contribute

- [ ] Yes, I want to contribute

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.