airbytehq / airbytehq/PyAirbyte

[source-slack] IncrementalMessageStream for threads channel never stops syncing

Abierto
#430 0 comentarios 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
Python
Estrellas
344
Forks
77
Merge medio
1 d 11 h
PR fusionados (30 d)
35

Descripción

### Connector Name

source-slack

### Connector Version

1.3.1

### What step the error happened?

During the sync

### Relevant information

Threads channel syncing never stops. It seems to cycle through all the channels, updating the end timestamp each loop (as some time passed while the data was queried), and does that over and over.
Other channels work fine.

Using PyAirbyte:

```python
import airbyte as ab

source = ab.get_source(
"source-slack",
config={
"api_token": "",
"start_date": "2024-01-01T00:00:00Z",
"lookback_window": 1,
"join_channels": False,
},
install_if_missing=True,
)
source.check()
source.select_streams(["channels", "channel_members", "channel_messages", "threads"])

result = source.read()

for name, records in result.streams.items():
print(f"Stream {name}: {len(list(records))} records")
for record in records:
print(record.get("text"))

```

### Relevant log output

```shell
2024-10-22 15:40:57 - INFO - Check succeeded
2024-10-22 15:41:02 - INFO - Starting syncing SourceSlack
2024-10-22 15:41:02 - INFO - Marking stream channels as STARTED
2024-10-22 15:41:02 - INFO - Setting state of SourceSlack stream to {'__ab_no_cursor_state_message': True}
2024-10-22 15:41:02 - INFO - Syncing stream: channels
2024-10-22 15:41:02 - INFO - Marking stream channels as RUNNING
2024-10-22 15:41:02 - INFO - Read 4 records from channels stream
2024-10-22 15:41:02 - INFO - Marking stream channels as STOPPED
2024-10-22 15:41:02 - INFO - Finished syncing channels
2024-10-22 15:41:02 - INFO - SourceSlack runtimes:
Syncing stream channels 0:00:00.365530
2024-10-22 15:41:02 - INFO - Marking stream channel_members as STARTED
2024-10-22 15:41:02 - INFO - Setting state of SourceSlack stream to {'__ab_no_cursor_state_message': True}
2024-10-22 15:41:02 - INFO - Syncing stream: channel_members
2024-10-22 15:41:02 - INFO - Marking stream channel_members as RUNNING
2024-10-22 15:41:03 - INFO - Read 18 records from channel_members stream
2024-10-22 15:41:03 - INFO - Marking stream channel_members as STOPPED
2024-10-22 15:41:03 - INFO - Finished syncing channel_members
2024-10-22 15:41:03 - INFO - SourceSlack runtimes:
Syncing stream channel_members 0:00:00.985201
Syncing stream channels 0:00:00.365530
2024-10-22 15:41:03 - INFO - Marking stream channel_messages as STARTED
2024-10-22 15:41:03 - INFO - Setting state of SourceSlack stream to {'states': [{'partition': {'channel': 'C07Q9KX2U6T', 'parent_slice': {}}, 'cursor': {'float_ts': 1729324525.094199}}, {'partition': {'channel': 'C07QHGM9B1C', 'parent_slice': {}}, 'cursor': {'float_ts': 1729324525.118059}}, {'partition': {'channel': 'C07RCUS2MGQ', 'parent_slice': {}}, 'cursor': {'float_ts': 1729324525.155319}}, {'partition': {'channel': 'C07RFRXUQN5', 'parent_slice': {}}, 'cursor': {'float_ts': 1729283939.911469}}]}
2024-10-22 15:41:03 - INFO - Syncing stream: channel_messages
2024-10-22 15:41:03 - INFO - Marking stream channel_messages as RUNNING
2024-10-22 15:41:04 - INFO - Read 6 records from channel_messages stream
2024-10-22 15:41:04 - INFO - Marking stream channel_messages as STOPPED
2024-10-22 15:41:04 - INFO - Finished syncing channel_messages
2024-10-22 15:41:04 - INFO - SourceSlack runtimes:
Syncing stream channel_members 0:00:00.985201
Syncing stream channel_messages 0:00:01.123485
Syncing stream channels 0:00:00.365530
2024-10-22 15:41:04 - INFO - Marking stream threads as STARTED
2024-10-22 15:41:04 - INFO - Setting state of SourceSlack stream to {}
2024-10-22 15:41:04 - INFO - Syncing stream: threads
2024-10-22 15:41:04 - INFO - Reading the channel: `test-slack-ingestion`
2024-10-22 15:41:04 - INFO - Syncing replies {'channel': 'C07Q9KX2U6T', 'oldest': 1704067200.0, 'latest': 1712707200.0}
2024-10-22 15:41:04 - INFO - Syncing replies {'channel': 'C07Q9KX2U6T', 'oldest': 1712707200.0, 'latest': 1721347200.0}
2024-10-22 15:41:05 - INFO - Syncing replies {'channel': 'C07Q9KX2U6T', 'oldest': 1721347200.0, 'latest': 1729600864.626089}
2024-10-22 15:41:05 - INFO - Marking stream threads as RUNNING
2024-10-22 15:41:07 - INFO - Reading the channel: `social`
2024-10-22 15:41:07 - INFO - Syncing replies {'channel': 'C07QHGM9B1C', 'oldest': 1704067200.0, 'latest': 1712707200.0}
2024-10-22 15:41:07 - INFO - Syncing replies {'channel': 'C07QHGM9B1C', 'oldest': 1712707200.0, 'latest': 1721347200.0}
2024-10-22 15:41:07 - INFO - Syncing replies {'channel': 'C07QHGM9B1C', 'oldest': 1721347200.0, 'latest': 1729600867.384261}
2024-10-22 15:41:08 - INFO - Reading the channel: `all-new-workspace`
2024-10-22 15:41:08 - INFO - Syncing replies {'channel': 'C07RCUS2MGQ', 'oldest': 1704067200.0, 'latest': 1712707200.0}
2024-10-22 15:41:09 - INFO - Syncing replies {'channel': 'C07RCUS2MGQ', 'oldest': 1712707200.0, 'latest': 1721347200.0}
2024-10-22 15:41:09 - INFO - Syncing replies {'channel': 'C07RCUS2MGQ', 'oldest': 1721347200.0, 'latest': 1729600868.85753}
2024-10-22 15:41:10 - INFO - Reading the channel: `support`
2024-10-22 15:41:10 - INFO - Syncing replies {'channel': 'C07RFRXUQN5', 'oldest': 1704067200.0, 'latest': 1712707200.0}
2024-10-22 15:41:10 - INFO - Syncing replies {'channel': 'C07RFRXUQN5', 'oldest': 1712707200.0, 'latest': 1721347200.0}
2024-10-22 15:41:10 - INFO - Syncing replies {'channel': 'C07RFRXUQN5', 'oldest': 1721347200.0, 'latest': 1729600870.328075}
2024-10-22 15:41:12 - INFO - Reading the channel: `test-slack-ingestion`
2024-10-22 15:41:12 - INFO - Syncing replies {'channel': 'C07Q9KX2U6T', 'oldest': 1704067200.0, 'latest': 1712707200.0}
2024-10-22 15:41:12 - INFO - Syncing replies {'channel': 'C07Q9KX2U6T', 'oldest': 1712707200.0, 'latest': 1721347200.0}
2024-10-22 15:41:12 - INFO - Syncing replies {'channel': 'C07Q9KX2U6T', 'oldest': 1721347200.0, 'latest': 1729600872.45988}
2024-10-22 15:41:14 - INFO - Reading the channel: `social`
2024-10-22 15:41:14 - INFO - Syncing replies {'channel': 'C07QHGM9B1C', 'oldest': 1704067200.0, 'latest': 1712707200.0}
2024-10-22 15:41:14 - INFO - Syncing replies {'channel': 'C07QHGM9B1C', 'oldest': 1712707200.0, 'latest': 1721347200.0}
2024-10-22 15:41:14 - INFO - Syncing replies {'channel': 'C07QHGM9B1C', 'oldest': 1721347200.0, 'latest': 1729600874.548036}
2024-10-22 15:41:15 - INFO - Reading the channel: `all-new-workspace`
2024-10-22 15:41:15 - INFO - Syncing replies {'channel': 'C07RCUS2MGQ', 'oldest': 1704067200.0, 'latest': 1712707200.0}
2024-10-22 15:41:15 - INFO - Syncing replies {'channel': 'C07RCUS2MGQ', 'oldest': 1712707200.0, 'latest': 1721347200.0}
2024-10-22 15:41:15 - INFO - Syncing replies {'channel': 'C07RCUS2MGQ', 'oldest': 1721347200.0, 'latest': 1729600875.567569}
2024-10-22 15:41:16 - INFO - Reading the channel: `support`
2024-10-22 15:41:16 - INFO - Syncing replies {'channel': 'C07RFRXUQN5', 'oldest': 1704067200.0, 'latest': 1712707200.0}
2024-10-22 15:41:16 - INFO - Syncing replies {'channel': 'C07RFRXUQN5', 'oldest': 1712707200.0, 'latest': 1721347200.0}
2024-10-22 15:41:16 - INFO - Syncing replies {'channel': 'C07RFRXUQN5', 'oldest': 1721347200.0, 'latest': 1729600876.609494}
2024-10-22 15:41:18 - INFO - Reading the channel: `test-slack-ingestion`
2024-10-22 15:41:18 - INFO - Syncing replies {'channel': 'C07Q9KX2U6T', 'oldest': 1704067200.0, 'latest': 1712707200.0}
2024-10-22 15:41:18 - INFO - Syncing replies {'channel': 'C07Q9KX2U6T', 'oldest': 1712707200.0, 'latest': 1721347200.0}
2024-10-22 15:41:18 - INFO - Syncing replies {'channel': 'C07Q9KX2U6T', 'oldest': 1721347200.0, 'latest': 1729600878.290911}
2024-10-22 15:41:20 - INFO - Reading the channel: `social`
2024-10-22 15:41:20 - INFO - Syncing replies {'channel': 'C07QHGM9B1C', 'oldest': 1704067200.0, 'latest': 1712707200.0}
2024-10-22 15:41:20 - INFO - Syncing replies {'channel': 'C07QHGM9B1C', 'oldest': 1712707200.0, 'latest': 1721347200.0}
2024-10-22 15:41:20 - INFO - Syncing replies {'channel': 'C07QHGM9B1C', 'oldest': 1721347200.0, 'latest': 1729600880.449153}
2024-10-22 15:41:21 - INFO - Reading the channel: `all-new-workspace`
2024-10-22 15:41:21 - INFO - Syncing replies {'channel': 'C07RCUS2MGQ', 'oldest': 1704067200.0, 'latest': 1712707200.0}
2024-10-22 15:41:21 - INFO - Syncing replies {'channel': 'C07RCUS2MGQ', 'oldest': 1712707200.0, 'latest': 1721347200.0}
2024-10-22 15:41:21 - INFO - Syncing replies {'channel': 'C07RCUS2MGQ', 'oldest': 1721347200.0, 'latest': 1729600881.485752}
2024-10-22 15:41:22 - INFO - Reading the channel: `support`
2024-10-22 15:41:22 - INFO - Syncing replies {'channel': 'C07RFRXUQN5', 'oldest': 1704067200.0, 'latest': 1712707200.0}
2024-10-22 15:41:22 - INFO - Syncing replies {'channel': 'C07RFRXUQN5', 'oldest': 1712707200.0, 'latest': 1721347200.0}
2024-10-22 15:41:22 - INFO - Syncing replies {'channel': 'C07RFRXUQN5', 'oldest': 1721347200.0, 'latest': 1729600882.548746}
2024-10-22 15:41:24 - INFO - Reading the channel: `test-slack-ingestion`
2024-10-22 15:41:24 - INFO - Syncing replies {'channel': 'C07Q9KX2U6T', 'oldest': 1704067200.0, 'latest': 1712707200.0}
2024-10-22 15:41:24 - INFO - Syncing replies {'channel': 'C07Q9KX2U6T', 'oldest': 1712707200.0, 'latest': 1721347200.0}
2024-10-22 15:41:24 - INFO - Syncing replies {'channel': 'C07Q9KX2U6T', 'oldest': 1721347200.0, 'latest': 1729600884.416193}
2024-10-22 15:41:27 - INFO - Reading the channel: `social`
2024-10-22 15:41:27 - INFO - Syncing replies {'channel': 'C07QHGM9B1C', 'oldest': 1704067200.0, 'latest': 1712707200.0}
2024-10-22 15:41:27 - INFO - Syncing replies {'channel': 'C07QHGM9B1C', 'oldest': 1712707200.0, 'latest': 1721347200.0}
2024-10-22 15:41:27 - INFO - Syncing replies {'channel': 'C07QHGM9B1C', 'oldest': 1721347200.0, 'latest': 1729600887.079985}
```

### Contribute

- [ ] Yes, I want to contribute

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.