taosdata / taosdata/TDengine

support for similar feature as kafka's max-poll-records in tdengine data subscription

Open
#29,792 1 comment 0 reactions 1 assignee View on GitHub

@yu285 is already working on this.

Since Feb 17, 2025.

enhancement
Dominant language
C
Stars
25.1k
Forks
5k
Avg merge
4d 59m
Merged PRs (30d)
7

Description

需求描述:
最近在使用TDengine替代kafka做数据订阅时遇到一个问题,使用ConsumerRecords records = consumer.poll(Duration.ofMillis(100));获取到数据的时候,会获取到很多数据,具体多少不清楚,但是我的消费者处理速度不高,并且有可能会停机更新,停机的时候要求把当前poll到的数据处理完毕,那如果poll到太多数据的话就会导致本次停机时间需要很长。如果我能控制每次拉取的最大数量,那么我就可以根据处理的数据灵活配置。我看了下TDengine关于消息订阅这块并没有类似kafka中max-poll-records这个参数,我尝试在创建TOPIC的时候SQL语句后面加上limit 5,但不行,语法错误。希望能加上这个配置参数,实现灵活设置poll拉取的最大消息数目,感谢
应用场景:
大部分的数据订阅场景
参考对象:
kafka https://kafka.apache.org/documentation/#consumerconfigs_max.poll.records

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.