apache / apache/beam

Support Redis Cluster when reading with RedisIO

Open
#21,124 5 comments 1 reaction 0 assignees View on GitHub
io java new feature P3 redis
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

I am trying to use the RedisIO connector with Redis cluster but it looks like the Jedis client that RedisIO uses only works on a standalone Redis server, not on a cluster. I get this error when trying to read from Redis:

```

Exception in thread "main" org.apache.beam.sdk.Pipeline$PipelineExecutionException: redis.clients.jedis.exceptions.JedisMovedDa
taException:
MOVED 15000 172.16.2.3:6379
at org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:371)

at org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:339)

at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:219)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)

at org.apache.beam.sdk.Pipeline.run(Pipeline.java:322)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:308)

at com.oracle.quanta.RedisToAtp.run(RedisToAtp.java:196)
at com.oracle.quanta.RedisToAtp.main(RedisToAtp.java:54)
Caused
by: redis.clients.jedis.exceptions.JedisMovedDataException: MOVED 15000 172.16.2.3:6379
at
redis.clients.jedis.Protocol.processError(Protocol.java:116)
at redis.clients.jedis.Protocol.process(Protocol.java:166)

at redis.clients.jedis.Protocol.read(Protocol.java:220)
at redis.clients.jedis.Connection.readProtocolWithCheckingBroken(Connection.java:278)

at redis.clients.jedis.Connection.getBinaryMultiBulkReply(Connection.java:230)
at redis.clients.jedis.Connection.getMultiBulkReply(Connection.java:224)

at redis.clients.jedis.Jedis.mget(Jedis.java:474)
at org.apache.beam.sdk.io.redis.RedisIO$ReadFn.fetchAndFlush(RedisIO.java:517)

at org.apache.beam.sdk.io.redis.RedisIO$ReadFn.finishBundle(RedisIO.java:500)

```

This is the code that I use:

```

PCollection events =

pipeline

/*


* Step #1: Read from Redis.

*/


.apply("Read Redis KV Store", RedisIO.read()

.withEndpoint(redisHost,
6379)

.withKeyPattern(redisKeyPattern))

```

Is there a way to configure RedisIO to work with a cluster? I would have expected it to use JedisCluster when working with Redis in cluster mode but from https://github.com/apache/beam/blob/master/sdks/java/io/redis/src/main/java/org/apache/beam/sdk/io/redis/RedisIO.java it appears that it only uses the standalone Jedis client.

Source: https://lists.apache.org/thread.html/rb2826a1b081055ed64ca56b7c8985fbe121488fea5bd5830a38a38f7%40%3Cuser.beam.apache.org%3E

Imported from Jira [BEAM-13065](https://issues.apache.org/jira/browse/BEAM-13065). Original Jira may contain additional context.
Reported by: lcwik.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.