apache / apache/hudi

[SUPPORT] Best Practices for Setting Up DeltaStreamer Jobs for PostgreSQL Partitioned Tables with Debezium

Open
#11,705 7 comments 0 reactions 0 assignees View on GitHub
area:ingest type:feature
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

Hi everyone,

I’m working with a PostgreSQL table that uses a hash-based partitioning strategy. Here is the setup:

```
-- Create the main sales table
CREATE TABLE public.sales
(
salesid SERIAL,
invoiceid integer,
itemid integer,
category text,
price numeric(10,2),
quantity integer,
orderdate date,
destinationstate text,
shippingtype text,
referral text,
updated_at TIMESTAMP DEFAULT NOW(),
PRIMARY KEY (salesid, invoiceid) -- Include the partitioning column in the primary key
) PARTITION BY HASH (invoiceid);

-- Create partitions
CREATE TABLE public.sales_part_0 PARTITION OF public.sales FOR VALUES WITH (MODULUS 4, REMAINDER 0);
CREATE TABLE public.sales_part_1 PARTITION OF public.sales FOR VALUES WITH (MODULUS 4, REMAINDER 1);
CREATE TABLE public.sales_part_2 PARTITION OF public.sales FOR VALUES WITH (MODULUS 4, REMAINDER 2);
CREATE TABLE public.sales_part_3 PARTITION OF public.sales FOR VALUES WITH (MODULUS 4, REMAINDER 3);

-- Insert data
INSERT INTO public.sales (invoiceid, itemid, category, price, quantity, orderdate, destinationstate, shippingtype, referral)
VALUES
(101, 1, 'Electronics', 599.99, 2, '2023-11-21', 'California', 'Express', 'Friend'),
(102, 3, 'Clothing', 49.99, 5, '2023-11-22', 'New York', 'Standard', 'OnlineAd'),
(103, 2, 'Home & Garden', 199.50, 1, '2023-11-23', 'Texas', 'Express', 'WordOfMouth'),
(104, 4, 'Books', 15.75, 3, '2023-11-24', 'Florida', 'Standard', 'SocialMedia'),
(105, 2, 'Home & Garden', 199.50, 1, '2023-11-23', 'Texas', 'Express', 'WordOfMouth');

```

I have set up a Debezium connector with the following configuration:

```
name=PostgresConnector
connector.class=io.debezium.connector.postgresql.PostgresConnector
tasks.max=1
database.user=hive
database.dbname=metastore
database.hostname=metastore_db
database.password=hive
database.server.name=hive
table.include.list=public.sales
database.port=5432
plugin.name=pgoutput
tombstones.on.delete=false

```
This configuration creates the following topics:
![image](https://github.com/user-attachments/assets/57ab5fd5-acbd-43c4-a6a0-7c9f952dc118)

hive.public.sales
hive.public.sales_part_0
hive.public.sales_part_1
hive.public.sales_part_2
hive.public.sales_part_3
I am looking for recommendations on how to set up DeltaStreamer jobs for each partition. What is the best approach to handle this setup effectively?

Thank you!

Contributor guide

No contributing guide indexed for this repository

Research direction

No repository file, test, or entry point is named. Start by reviewing the PostgreSQL partitioning and Debezium configuration shown here, then determine how DeltaStreamer should consume the listed partition topics. Done means documenting a clear recommended job setup and how all partitions are handled.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql
Domain
data-engineering, databases, stream-processing
Issue type
Documentation
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.