apache / apache/beam

[Bug]: RedisIO#readKeyPatterns failing with OutOfMemory on version 2.39.0

Open
#21,825 11 comments 0 reactions 0 assignees View on GitHub
bug io java P2 redis
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

### What happened?

I'm using **RedisIO** to read/write values on cache and it was working fine with version _2.38.0._

When I moved to version _2.39.0_ I started getting errors like:

`Execution of work for computation 'P13' for key '<�֤#�͵.;]�NDC���^m��=�w<@癢P�u:k<��ުm�7����6�';� �z �< z0��>2QN����9EX�-����k\����eL� �( ���a����;��φ' failed with out-of-memory. Work will not be retried locally. Heap dump not written.`

With stacktrace:

```
java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
at java.base/java.lang.Thread.start0(Native Method)
at java.base/java.lang.Thread.start(Thread.java:803)
at java.base/java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:937)
at java.base/java.util.concurrent.ThreadPoolExecutor.ensurePrestart(ThreadPoolExecutor.java:1583)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:346)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:562)
at java.base/java.util.concurrent.Executors$DelegatedScheduledExecutorService.schedule(Executors.java:779)
at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.OutputAndTimeBoundedSplittableProcessElementInvoker$ProcessContext.onClaimed(OutputAndTimeBoundedSplittableProcessElementInvoker.java:312)
at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.sdk.fn.splittabledofn.RestrictionTrackers$RestrictionTrackerObserver.tryClaim(RestrictionTrackers.java:60)
at org.apache.beam.sdk.io.redis.RedisIO$ReadFn.processElement(RedisIO.java:399)
```

The error happens when calling `RedisIO.readKeyPatterns()`

The pipeline fails and gets stuck in the Step: `.../ReadRedis/ParDo(Read)/ParMultiDo(Read)/ProcessKeyedElements.out0`

When I move back to version _2.38.0_ the problem no longer happens

### Issue Priority

Priority: 2

### Issue Component

Component: io-java-redis

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.