ydb-platform / ydb-platform/ydb-java-sdk

SyncWriter and flush problem throwing Exception

Open
#475 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Java
Stars
53
Forks
40
Avg merge
9h 51m
Merged PRs (30d)
7

Description

When writing to topic code may produce a RuntimeException wraped in CompletionException.

For example, writing with codec = 10000 or 7.


 WriterSettings settings = WriterSettings.newBuilder()
                .setTopicPath(topicName)
                .setCodec(7)
                .build();
        SyncWriter writer = client.createSyncWriter(settings);
         writer.initAndWait();


        for (byte[] testMessage : testMessages) {
            writer.send(Message.newBuilder().setData(testMessage).build());
        }

        writer.flush();
        writer.shutdown(1, TimeUnit.MINUTES);

writer.flush(); - is the cause of our problem
Correct behavior—work without exception because we wait for all jobs to be completed and return results;

But if you machine is too quick - you will get CompletionException

Spoiler warning java.util.concurrent.CompletionException: java.lang.RuntimeException: Unsupported codec: 7 at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273) ~[?:1.8.0_44] at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280) ~[?:1.8.0_44] at java.util.concurrent.CompletableFuture$AsyncRun.run$$$capture(CompletableFuture.java:1621) ~[?:1.8.0_44] at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java) ~[?:1.8.0_44] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_44] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_44] at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_44] Caused by: java.lang.RuntimeException: Unsupported codec: 7 at tech.ydb.topic.utils.Encoder.getCodec(Encoder.java:96) ~[classes/:?] at tech.ydb.topic.utils.Encoder.makeOutputStream(Encoder.java:80) ~[classes/:?] at tech.ydb.topic.utils.Encoder.encode(Encoder.java:41) ~[classes/:?] at tech.ydb.topic.write.impl.WriterImpl.encode(WriterImpl.java:195) ~[classes/:?] at tech.ydb.topic.write.impl.WriterImpl.lambda$acceptMessageIntoSendingQueue$0(WriterImpl.java:176) ~[classes/:?] at tech.ydb.topic.write.impl.WriterImpl$$Lambda$189/2108708444.run(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$AsyncRun.run$$$capture(CompletableFuture.java:1618) ~[?:1.8.0_44] ... 4 more 11:15:44.949 [pool-4-thread-2] ERROR tech.ydb.topic.write.impl.WriterImpl - [yNNtF0] Exception while encoding message: java.util.concurrent.CompletionException: java.lang.RuntimeException: Unsupported codec: 7 at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273) ~[?:1.8.0_44] at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280) ~[?:1.8.0_44] at java.util.concurrent.CompletableFuture$AsyncRun.run$$$capture(CompletableFuture.java:1621) ~[?:1.8.0_44] at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java) ~[?:1.8.0_44] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_44] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_44] at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_44] Caused by: java.lang.RuntimeException: Unsupported codec: 7 at tech.ydb.topic.utils.Encoder.getCodec(Encoder.java:96) ~[classes/:?] at tech.ydb.topic.utils.Encoder.makeOutputStream(Encoder.java:80) ~[classes/:?] at tech.ydb.topic.utils.Encoder.encode(Encoder.java:41) ~[classes/:?] at tech.ydb.topic.write.impl.WriterImpl.encode(WriterImpl.java:195) ~[classes/:?] at tech.ydb.topic.write.impl.WriterImpl.lambda$acceptMessageIntoSendingQueue$0(WriterImpl.java:176) ~[classes/:?] at tech.ydb.topic.write.impl.WriterImpl$$Lambda$189/2108708444.run(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$AsyncRun.run$$$capture(CompletableFuture.java:1618) ~[?:1.8.0_44] ... 4 more 11:15:44.950 [pool-4-thread-5] ERROR tech.ydb.topic.write.impl.WriterImpl - [yNNtF0] Exception while encoding message: java.util.concurrent.CompletionException: java.lang.RuntimeException: Unsupported codec: 7 at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273) ~[?:1.8.0_44] at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280) ~[?:1.8.0_44] at java.util.concurrent.CompletableFuture$AsyncRun.run$$$capture(CompletableFuture.java:1621) ~[?:1.8.0_44] at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java) ~[?:1.8.0_44] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_44] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_44] at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_44] Caused by: java.lang.RuntimeException: Unsupported codec: 7 at tech.ydb.topic.utils.Encoder.getCodec(Encoder.java:96) ~[classes/:?] at tech.ydb.topic.utils.Encoder.makeOutputStream(Encoder.java:80) ~[classes/:?] at tech.ydb.topic.utils.Encoder.encode(Encoder.java:41) ~[classes/:?] at tech.ydb.topic.write.impl.WriterImpl.encode(WriterImpl.java:195) ~[classes/:?] at tech.ydb.topic.write.impl.WriterImpl.lambda$acceptMessageIntoSendingQueue$0(WriterImpl.java:176) ~[classes/:?] at tech.ydb.topic.write.impl.WriterImpl$$Lambda$189/2108708444.run(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$AsyncRun.run$$$capture(CompletableFuture.java:1618) ~[?:1.8.0_44] ... 4 more 11:15:44.950 [pool-4-thread-3] ERROR tech.ydb.topic.write.impl.WriterImpl - [yNNtF0] Exception while encoding message: java.util.concurrent.CompletionException: java.lang.RuntimeException: Unsupported codec: 7 at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273) ~[?:1.8.0_44] at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280) ~[?:1.8.0_44] at java.util.concurrent.CompletableFuture$AsyncRun.run$$$capture(CompletableFuture.java:1621) ~[?:1.8.0_44] at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java) ~[?:1.8.0_44] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_44] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_44] at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_44] Caused by: java.lang.RuntimeException: Unsupported codec: 7 at tech.ydb.topic.utils.Encoder.getCodec(Encoder.java:96) ~[classes/:?] at tech.ydb.topic.utils.Encoder.makeOutputStream(Encoder.java:80) ~[classes/:?] at tech.ydb.topic.utils.Encoder.encode(Encoder.java:41) ~[classes/:?] at tech.ydb.topic.write.impl.WriterImpl.encode(WriterImpl.java:195) ~[classes/:?] at tech.ydb.topic.write.impl.WriterImpl.lambda$acceptMessageIntoSendingQueue$0(WriterImpl.java:176) ~[classes/:?] at tech.ydb.topic.write.impl.WriterImpl$$Lambda$189/2108708444.run(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$AsyncRun.run$$$capture(CompletableFuture.java:1618) ~[?:1.8.0_44] ... 4 more

After some research I can make test work more stable without throwing Exception(May be I was lucking). It's just a guess and should be checked:

  1. WriterImpl - make lastAcceptedMessageFuture volatile. (Seems it's changed in different threads)
  2. WriterImpl - acceptMessageIntoSendingQueue should return CompletableFuture
  3. WriterImpl - tryToEnqueue should return acceptMessageIntoSendingQueue(message);

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in WriterImpl at acceptMessageIntoSendingQueue, tryToEnqueue, and lastAcceptedMessageFuture, then trace Encoder.getCodec using the supplied codec 7 or 10000 reproduction. Run the flush scenario and verify that completion and failure behavior is deterministic and matches the issue's expected result.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.