apache / apache/beam

[Feature Request]: Create Apache Beam PostgreSQL I/O Connector with streaming support

Open
#39,959 1 comment 0 reactions 1 assignee Claimed by @jk-shah View on GitHub
io new feature P2
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
205

Description

### What would you like to happen?

**Feature**: Support a native PostgreSQL I/O with streaming support.

**Expectation**: Create a a dedicated, high-performance PostgreSQL connector with native PostgreSQL streaming and write sinks that works with databases that are either self-hosted or managed.

**Requirements**: Main functionality will be to deliver the initial snapshot and stream continuity. It should be able to support native PostgreSQL binary protocol and handle transactional and non-transactional logical messages. Ability to support two-phase commit transactions. It must be able to handle TOAST columns in Beam runner-managed state without issuing point queries to the live database. Handling DDL changes in logical replication is hard but should be able to handle DDL changes or have guardrails around it. Finally it needs to do it at scale even for write sink to PostgreSQL.

### Issue Priority

Priority: 2 (default / most feature requests should be filed as P2)

### Issue Components

- [ ] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [x] Component: IO connector
- [ ] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Infrastructure
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Prism Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.