apache / apache/fluss

[filesystem] HadoopDataInputStream.seek loops forever past EOF

Open Beginner friendly
#3,718 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.

I searched titles, bodies, and comments for `HadoopDataInputStream`, `skipFully`, `seek past EOF`, `HDFS seek hang`, `Hadoop skip EOF`, and related filesystem terms. I also inspected the open Hadoop dependency-alignment PR #3699; it does not touch this code path.

### Fluss version

- `main (development)` at `31621117db9b21e8e00a6e716b3a13b7b66e18be`
- The same implementation is present in `v0.9.1-incubating`, `v0.9.0-incubating`, and `v0.8.0-incubating`

### Please describe the bug 🐞

`HadoopDataInputStream.seek(long)` optimizes small forward seeks by calling `skipFully(delta)`. If the requested position is past EOF, the underlying `InputStream.skip(long)` returns `0`. The loop in `skipFully` then makes no progress and spins forever:

https://github.com/apache/fluss/blob/31621117db9b21e8e00a6e716b3a13b7b66e18be/fluss-filesystems/fluss-fs-hadoop/src/main/java/org/apache/fluss/fs/hdfs/HadoopDataInputStream.java#L132-L135

This contradicts the public `FSDataInputStream.seek` contract, which says callers cannot seek past the end of the stream and reports seek errors through `IOException`:

https://github.com/apache/fluss/blob/31621117db9b21e8e00a6e716b3a13b7b66e18be/fluss-common/src/main/java/org/apache/fluss/fs/FSDataInputStream.java#L36-L43

Java's `InputStream.skip` contract permits returning fewer bytes than requested, including `0`.

#### Reproduction

The existing `SeekableByteArrayInputStream` test fixture already returns `0` from `skip` at EOF. Adding this focused case to `HadoopDataInputStreamTest` reproduces the hang:

```java
@Test
void testSeekPastEndOfStream() {
FSDataInputStream input =
new FSDataInputStream(new SeekableByteArrayInputStream(new byte[1]));
HadoopDataInputStream stream = new HadoopDataInputStream(input);

assertThatThrownBy(() -> stream.seek(2)).isInstanceOf(EOFException.class);
}
```

Run it with:

```bash
./mvnw -pl fluss-filesystems/fluss-fs-hadoop -am \
-Dtest=HadoopDataInputStreamTest \
-Dsurefire.failIfNoSpecifiedTests=false test
```

I also reproduced this twice against Hadoop's real local filesystem implementation using a one-byte file and `seek(2)`. Both runs timed out after two seconds:

```text
before seek target=2 length=1
TIMEOUT after 2s (seek did not return)
```

The nearby control `seek(1)` returned normally with position `1`.

#### Expected behavior

The call should promptly throw `IOException`/`EOFException` because the requested position is past EOF.

#### Actual behavior

`seek(2)` never returns. The thread remains in the `skipFully` loop.

No special configuration or external service is required.

### Solution

Delegate `skipFully` to Hadoop's existing `org.apache.hadoop.io.IOUtils.skipFully(InputStream, long)`, or implement the same zero-progress handling: when `skip` returns `0`, read one byte and throw `EOFException` if EOF has been reached.

A regression test can be added to the existing `HadoopDataInputStreamTest`. This should require no new dependency and no API or storage-format change.

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with fluss-filesystems/fluss-fs-hadoop/src/main/java/org/apache/fluss/fs/hdfs/HadoopDataInputStream.java, focusing on seek and skipFully, then read HadoopDataInputStreamTest and the existing SeekableByteArrayInputStream fixture. Run the provided Maven test command and add the seek-past-EOF regression case. Done means seek(2) on a one-byte stream promptly throws EOFException or IOException while the nearby seek(1) behavior remains valid.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
84/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.