apache / apache/beam

[Feature Request]: Support random access in the Go Filesystem API (io.ReaderAt / io.ReadSeeker)

Open
#39,176 1 comment 1 reaction 0 assignees View on GitHub
awaiting triage go new feature P2
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

### What would you like to happen?

The Go Filesystem API should support random access when the underlying filesystem provides it.

Currently, `filesystem.Interface.OpenRead` returns an `io.ReadCloser`, which only supports sequential reads. This makes it difficult to implement readers for file formats such as Parquet, ORC, Arrow IPC, and similar formats that rely on seeking to arbitrary offsets within a file.

Although storage backends such as GCS, S3, and local files support range reads or random access, that capability is not exposed through the Go Filesystem abstraction. As a result, implementations either need to buffer the entire file into memory or bypass the Beam Filesystem API and use storage-specific clients directly.

It would be useful if the Go Filesystem API exposed a random access interface, for example by returning an `io.ReaderAt`, `io.ReadSeeker`, or another abstraction that supports efficient seeking when the underlying filesystem is capable of it.

This would make it easier to build portable, efficient Beam I/O connectors and readers for columnar and indexed file formats without sacrificing performance or bypassing the Beam filesystem abstraction.

### Issue Priority

Priority: 2 (default / most feature requests should be filed as P2)

### Issue Components

- [ ] Component: Python SDK
- [ ] Component: Java SDK
- [x] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Infrastructure
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Prism Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner

Contributor guide

Open the contributing guide

Research direction

Start with the Go SDK's filesystem.Interface.OpenRead entry point and trace the existing filesystem abstraction and implementations. Compare the proposed io.ReaderAt and io.ReadSeeker options, then define an API that exposes efficient random access where supported without requiring storage-specific clients; done should include coverage for the affected filesystem behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
api
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.