apache / apache/cloudberry

[Bug] kafka_fdw foreign table returns zero rows: Feature not supported: Foreign Data

Open
#1,060 2 comments 0 reactions 0 assignees View on GitHub
type: Bug
Dominant language
C
Stars
1.4k
Forks
247
Avg merge
4d 3h
Merged PRs (30d)
39

Description

### Apache Cloudberry version

PostgreSQL 14.4 (Cloudberry Database 1.0.0 build dev) on x86_64-pc-linux-gnu, compiled by gcc (GCC) 11.5.0 20240719 (Red Hat 11.5.0-5), 64-bit compiled on Mar 24 2025 12:34:11 (with assert checkin
g)

### What happened

Description:

I have successfully created and configured the kafka_fdw extension and a foreign table to read from a Kafka topic. However, when querying the foreign table, it returns zero rows with the following log message:

`NOTICE: "Feature not supported: Foreign Data"`

This seems to indicate the foreign table was created correctly but data retrieval is blocked or unsupported by the executor.

Environment:

OS: Docker container

Kafka: 5.5.0 (also tried 7.4.0)

kafka_fdw: latest build from GitHub

Database: CloudberryDB / GreenplumDB

FDW server created with broker localhost:9092

Additional Context:

Kafka topic contrib_regress is successfully created. Using command-line tools, I can publish and consume messages. Seems like the FDW is skipping execution due to unsupported features.

Please advise on how to enable execution or whether a fix is available.

### What you think should happen instead

I think cloudberry has to get data from foreign database system.

### How to reproduce

Steps to Reproduce:

Installed and enabled kafka_fdw:

`CREATE EXTENSION kafka_fdw;`

Set up Kafka/Zookeeper using Docker:

version: '2'
networks:
test:
services:
zookeeper:
image: confluentinc/cp-zookeeper:5.5.0
ports: [ "2181:2181" ]
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
kafka:
image: confluentinc/cp-kafka:5.5.0
ports: [ "9092:9092" ]
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://192.168.222.51:9092
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

Created server and foreign table:

CREATE SERVER kafka_server
FOREIGN DATA WRAPPER kafka_fdw
OPTIONS (brokers 'localhost:9092');

CREATE USER MAPPING FOR PUBLIC SERVER kafka_server;
CREATE USER MAPPING FOR gpadmin SERVER kafka_server;

CREATE FOREIGN TABLE kafka_test (
part int OPTIONS (partition 'true'),
offs bigint OPTIONS (offset 'true'),
some_int int,
some_text text,
some_date date,
some_time timestamp
)
SERVER kafka_server
OPTIONS (format 'csv', topic 'contrib_regress', batch_size '30', buffer_delay '100');

Ran query:

`SELECT * FROM kafka_test LIMIT 1;`

Expected Behavior:
Should return rows from the Kafka topic contrib_regress.

Actual Behavior:
Returns zero rows with notice:

`NOTICE: "Feature not supported: Foreign Data"`

### Operating System

OS: Docker container Linux mdw 4.18.0-553.el8_10.x86_64 #1 SMP Fri May 24 13:05:10 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

### Anything else

_No response_

### Are you willing to submit PR?

- [ ] Yes, I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/cloudberry/blob/main/CODE_OF_CONDUCT.md).

Contributor guide

Open the contributing guide

Research direction

Start by reproducing SELECT * FROM kafka_test LIMIT 1 against the kafka_fdw foreign table and inspect the foreign-table execution path that emits “Feature not supported: Foreign Data.” Compare the Cloudberry executor’s handling of this query with kafka_fdw’s expected retrieval behavior; done means rows from the contrib_regress Kafka topic are returned without that notice.

Written by the indexing model from the issue text.

Assessment

Tech stack
kafka, postgresql
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.