GoogleCloudPlatform / GoogleCloudPlatform/gcs-analytics-core

Integrating gcs-analytics-core into Apache Hudi's native reader

Open
#334 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
13
Forks
18
Avg merge
5d 1h
Merged PRs (30d)
4

Description

### Context

We're prototyping the use of `gcs-analytics-core` to accelerate reads in [Apache Hudi](https://github.com/apache/hudi). Unlike engines that read Parquet through the Hadoop `FileSystem`, Hudi's highest-value, self-owned read path is its **native HFile reader** behind the metadata table (record-level index, bloom filters, column stats). A metadata point-lookup is a merge over `(1 base + D log)` HFiles with one `seekTo` per key, so it is dominated by per-request round-trip latency. We want to validate that we're using the library's APIs the way you intend before taking this further.

### Goals

- Route Hudi's native HFile reads through `GoogleCloudStorageInputStream` to get vectored I/O + prefetch/caching.
- Keep the integration behind a cloud-agnostic capability interface (`RangeReadable`) in Hudi, with gcs-analytics-core as the first implementation; graceful fallback when absent.
- Batch the data-block reads for a looked-up key set into a single vectored read instead of N serial `seek`+`read`.

### Non-goals

- No changes to base-Parquet reads (those come from the GCS Hadoop connector + parquet-hadoop
`readVectored`).
- No coupling of Hudi core to GCS; the dependency is optional / reflectively isolated.

### High-level design

Hudi gets a small `RangeReadable` mixin (positional `readFully` + `readVectored(List, IntFunction)`, where `FileRange = {offset, length, CompletableFuture}`). The GCS implementation wraps `GoogleCloudStorageInputStream`:

```java
// Construction (once per Hudi FileSystem / reader-factory instance)
GcsFileSystemOptions opts =
new GcsAnalyticsCoreOptions("fs.gs.", confAsMap).getGcsFileSystemOptions();
GcsFileSystem gcsFs = new GcsFileSystemImpl(opts);

// Per file
GoogleCloudStorageInputStream in = GoogleCloudStorageInputStream.create(gcsFs, uri);

// Positional read (must not move the stream cursor)
in.readFully(position, buffer, offset, length);

// Vectored: one call for all data blocks a key batch needs
List ranges = new ArrayList<>();
for (FileRange r : hudiRanges) {
ranges.add(GcsObjectRange.builder()
.setOffset(r.getOffset())
.setLength(r.getLength())
.setByteBufferFuture(r.getData()) // library completes our future
.build());
}
in.readVectored(ranges, ByteBuffer::allocate);
// consumer waits on each r.getData(), copies bytes out, parses the HFile block
```

The native HFile reader maps the sorted key set to distinct data blocks (via its in-memory block index), issues one `readVectored`, and serves subsequent per-key seeks from the prefetched blocks.

### API-usage questions we'd like validation on

1. **Lifecycle & sharing**: Is one `GcsFileSystemImpl` per Hadoop `FileSystem` / reader-factory instance the intended model? Is it safe to open many `GoogleCloudStorageInputStream`s from it, and to use them, concurrently across threads?
2. **Avoiding a metadata RPC**: `create(fs, URI)` calls `getFileInfo` (an RPC). When Hudi already knows the object size, is `create(fs, GcsFileInfo)` the right way to skip it, and what is the minimal `GcsFileInfo` / `GcsItemInfo` we must construct?
3. **`readVectored` buffer contract**: When a range's future completes, what are the `ByteBuffer`'s `position`/`limit` (i.e. is it flipped, ready to read)? Are the buffers pooled/reused, or else must we copy out before the next operation?
4. **Coalescing ownership**: Should we hand you the raw (un-merged) block ranges and let the library coalesce via `analytics-core.read.vectored.range.merge-gap.max-bytes` / `...merged-size.max-bytes`, or pre-merge ourselves? Are ranges required to be sorted / non-overlapping, and is there a max count per call?
5. **Footer prefetch vs HFile**: Is `analytics-core.footer.prefetch.*` a **format-agnostic tail prefetch** (last N bytes near EOF), or is it Parquet/ORC-structure-aware? This determines whether we still need our own load-on-open optimization on GCS.
6. **Caching coordination**: Hudi buffers small metadata files fully in memory itself. Does enabling `small-file.cache` double-buffer, and would you recommend disabling it for our case and relying on `readVectored` + footer prefetch for large (streamed) files only?
7. **Config prefix**: We currently reuse the `fs.gs.` prefix (shared with the Hadoop connector). Is that appropriate, or do you recommend a distinct prefix so we don't inherit/mixup connector settings?

Thanks for open-sourcing this! Guidance on the above would help us converge on an idiomatic integration.

Contributor guide

Open the contributing guide

Research direction

Start with the GoogleCloudStorageInputStream, GcsFileSystemImpl, GcsAnalyticsCoreOptions, GcsFileInfo, GcsItemInfo, GcsObjectRange, and readVectored APIs named in the issue. Review Hudi's native HFile metadata-table reader and RangeReadable proposal, then resolve the seven lifecycle, buffer, coalescing, caching, prefetch, and configuration questions with maintainers. Done means an agreed, cloud-agnostic integration design with optional fallback.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, java
Domain
backend-api-design, cloud
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.