Automatic Detection and Snapshotting of new tables on connector startup
- Dominant language
- HTML
- Stars
- 6
- Forks
- 8
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 1
Description
# Description
Currently, when users add new tables to the `table.include.list` regex pattern, they could need to manually trigger an incremental snapshot to capture the initial data. This enhancement proposes automating this process by tracking which tables have been snapshotted and intelligently detecting new tables on connector startup.
# Proposed Solution
Introduce a connector initialization hook that:
Receives connector config, connection, and offset information at startup and automatically detects tables matching the regex that haven't been snapshotted yet
Queues snapshot signals (blocking or incremental) for execution during startup
Separates snapshot state tracking from offset storage (similar to schema history topic approach)
This provides extensibility for both blocking and incremental snapshot strategies while keeping the codebase flexible for future enhancements.
# Suggested Subtasks
## Phase 1
- [ ] Design Debezium state storage mechanism - Define a separate store for Debezium state to contain information about the state of Debezium that are not strict related to the source processing. Initially it will be used to store snapshot metadata (similar to schema history topic) to avoid bloating the offset state. In future we can think to move other things currently in the offset in this dedicated store.
## Phase 2
- [ ] Implement startup initialization hook - Create the connector initialization method that receives config, connection, and offset/Debezium state data
- [ ] Implement blocking snapshot strategy - Auto-detect new tables on startup and queue blocking snapshots
- [ ] Add per-table snapshot tracking - Track which tables have been snapshotted to enable accurate diff detection on startup
## Phase 3
- [ ] Implement incremental snapshot strategy - Enable parallel streaming with incremental snapshotting for new tables
Contributor guide
Assessment
This issue has not been assessed yet.