support for similar feature as kafka's max-poll-records in tdengine data subscription
Open
@yu285 is already working on this.
Since Feb 17, 2025.
enhancement
- Dominant language
- C
- Stars
- 25.1k
- Forks
- 5k
- Avg merge
- 4d 59m
- Merged PRs (30d)
- 7
Description
需求描述:
最近在使用TDengine替代kafka做数据订阅时遇到一个问题,使用ConsumerRecords records = consumer.poll(Duration.ofMillis(100));获取到数据的时候,会获取到很多数据,具体多少不清楚,但是我的消费者处理速度不高,并且有可能会停机更新,停机的时候要求把当前poll到的数据处理完毕,那如果poll到太多数据的话就会导致本次停机时间需要很长。如果我能控制每次拉取的最大数量,那么我就可以根据处理的数据灵活配置。我看了下TDengine关于消息订阅这块并没有类似kafka中max-poll-records这个参数,我尝试在创建TOPIC的时候SQL语句后面加上limit 5,但不行,语法错误。希望能加上这个配置参数,实现灵活设置poll拉取的最大消息数目,感谢
应用场景:
大部分的数据订阅场景
参考对象:
kafka https://kafka.apache.org/documentation/#consumerconfigs_max.poll.records
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.