GreptimeTeam / GreptimeTeam/greptimedb-ingester-java

BulkStreamWriter does not recover after TCP connection reset

Open
#104 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
4
Forks
1
PR merge metrics
No merged PRs in 30d

Description

## Problem

`BulkStreamWriter` detects a broken TCP connection and exposes a `FlightRuntimeException`, but it does not recreate the underlying Arrow Flight `DoPut` stream. Once the stream becomes terminal, the writer remains unusable even when the endpoint is reachable again. The terminal state is permanent in the current implementation because the completion future cannot be reset and `startPut()` is never called again.

Applications must currently discard the failed writer and create a new one manually. At the protocol level, reconnecting a transport cannot revive an already-terminal bidirectional RPC; a new `DoPut` RPC is required.

The reproduction also observed an Arrow allocator leak error while closing the failed writer. This may be a cleanup interaction in the test harness and needs separate confirmation.

## Reproduction

Environment:

- Client commit: `37fc57cf9e01d3695d9c8e0732461d919e9578a9`
- GreptimeDB endpoint: `localhost:24001`
- GreptimeDB version: unknown; endpoint was provided through a local `k9s` port-forward
- Java: OpenJDK 21.0.10
- OS: Ubuntu 24.04, Linux 6.17.0-22-generic, x86_64
- Arrow Flight: 14.0.2
- gRPC: 1.78.0
- gRPC transport observed in the stack trace: Netty NIO
- Client settings: route refresh disabled, per-message timeout 5 seconds, max in-flight requests 4, zero-copy enabled

The reproduction used a disposable TCP proxy between the client and `localhost:24001`:

1. Use the regular write API to create a test table.
2. Create a client whose endpoint is the local TCP proxy.
3. Create one `BulkStreamWriter` and write one batch successfully.
4. Configure `SO_LINGER=0` and close both active proxy sockets to inject a TCP RST while keeping the proxy listener available.
5. Wait one second and write another batch with the same writer.
6. Create a replacement `BulkStreamWriter` through the same proxy and write another batch.

The relevant client calls were:

```java
assertEquals(1, write(writer, "before-reset", 1.0));
proxy.resetActiveConnections();
Thread.sleep(1000);

// Throws FlightRuntimeException: UNAVAILABLE: io exception.
write(writer, "after-reset", 2.0);

BulkStreamWriter replacement = client.bulkStreamWriter(schema, config);
assertEquals(1, write(replacement, "replacement-writer", 3.0));
```

`write(...)` creates a new `TableBufferRoot`, adds one row, calls `complete()`, and waits on `writer.writeNext().get(10, TimeUnit.SECONDS)`.

Command used for the integration harness:

```shell
mvn -pl ingester-integration-tests -am \
-Dit.test=BulkWriteReconnectIT \
-DargLine="--add-opens=java.base/java.nio=ALL-UNNAMED" \
verify
```

Relevant output:

```text
Caused by: java.net.SocketException: Connection reset

REPRODUCED: existing writer failed after TCP reset;
acceptedConnectionsBeforeReset=1,
acceptedConnectionsNow=1,
errorType=org.apache.arrow.flight.FlightRuntimeException,
error=org.apache.arrow.flight.FlightRuntimeException: UNAVAILABLE: io exception

REPRODUCED: replacement writer succeeded; acceptedConnections=2

Memory was leaked by query. Memory leaked: (16)
Allocator(BufferAllocator(Location{uri=grpc+tcp://localhost:})) ...
```

The harness passed because it asserted the current behavior: the original writer fails and a manually created replacement succeeds. Both successful writes returned an affected-row count of 1. Persistence was not independently queried because the port-forwarded instance's SQL endpoint required credentials unavailable to this harness.

The reset was injected between batches. This reproduction therefore demonstrates failure to recover the writer for later batches; it does not establish the server-side outcome of a batch interrupted while in flight.

## Current implementation

- `BulkWriteClient.bulkStreamWriteTo()` routes once and creates one `BulkWriteManager`: `ingester-protocol/src/main/java/io/greptime/BulkWriteClient.java`
- `BulkWriteService` calls `manager.startPut(...)` only in its constructor and stores the resulting listener permanently: `ingester-bulk-protocol/src/main/java/io/greptime/BulkWriteService.java`
- A transport error reaches `AsyncPutListener.onError()` and completes tracked in-flight futures exceptionally.
- There is no retry, reroute, manager replacement, channel replacement, or `startPut()` re-invocation.
- `BulkFlightClient` can observe a terminal stream error, but it cannot recreate the `DoPut` RPC: `ingester-bulk-protocol/src/main/java/org/apache/arrow/flight/BulkFlightClient.java`

## Expected behavior

After a terminal transport failure, the current batch should fail with the transport error. A later `writeNext()` should create a new `DoPut` stream automatically when the endpoint is reachable again, without replaying the failed batch implicitly.

Handling the batch that was in flight at disconnect requires care: its server-side outcome may be unknown, so automatic replay can introduce duplicates unless an idempotency mechanism exists.

At minimum:

- Mark the writer terminal immediately and consistently.
- Do not allow later writes to hang or degrade into per-message timeout.
- Do not leak limiter permits or Arrow allocator memory.
- Expose enough state/error information for callers to recreate the writer safely.
- Document whether unacknowledged batches can be replayed.

## Suggested tests

- Successful batch followed by TCP RST.
- In-flight futures receive the terminal stream error.
- Writes after terminal failure fail immediately and consistently.
- Recovery creates a new `DoPut` stream; whether the implementation also replaces the manager or channel is an implementation choice.
- Failed-writer close releases all Arrow memory.
- Concurrent write racing with stream termination does not miss error fan-out.
- Limiter permits are released when `putNext()` throws synchronously.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with BulkWriteClient.java, BulkWriteService.java, and BulkFlightClient.java to trace manager creation, startPut(), and terminal error handling. Run the stated BulkWriteReconnectIT Maven command, then use the suggested reset, in-flight, concurrency, close, and limiter cases to define the current failure behavior. Done means later writes fail or recover consistently without replaying an unacknowledged batch, leaked permits, or Arrow memory.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend-api-design, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.