redpanda-data / redpanda-data/connect

Add batching to redis_hash output to improve performance

Open
#3,771 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

ux
Dominant language
Go
Stars
8.8k
Forks
969
Avg merge
1d 13h
Merged PRs (30d)
64

Description

I am starting to experience performance problems when using the redis_hash output at a high number operations/s. The current implementation of the redis_hash output does not utilize the pipelining feature of redis-go. Adding this, similar to other redis outputs might be a good idea.

I have little experience with go, but I am willing to attempt this, staying close to the original implementation and borrowing parts of the pipelining implementation of the redis-list output.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the redis_hash output, then compare it with the redis-list output's pipelining implementation and inspect how redis-go supports pipelining. Add batching to redis_hash while staying close to its current implementation, and verify that the output uses pipelining as intended.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, redis
Domain
backend, stream-processing
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.