apache / apache/beam

GCSFileSystem requires gcp extra at lookup time while S3FileSystem does not

Open
#37,445 22 comments 0 reactions 1 assignee Claimed by @MansiSingh17 View on GitHub
good first issue
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

There's an inconsistency in how `FileSystems.get_filesystem()` handles missing optional dependencies between GCS and S3.

### Current Behavior

**S3 (without `aws` extra):**
```python
>>> from apache_beam.io import filesystems
>>> filesystems.FileSystems.get_filesystem("s3://blah")

```
Returns the filesystem object; validation happens later when the filesystem is actually used.

**GCS (without `gcp` extra):**
```python
>>> from apache_beam.io import filesystems
>>> filesystems.FileSystems.get_filesystem("gcs://blah")
ValueError: Unable to get filesystem from specified path, please use the correct path or ensure the required dependency is installed, e.g., pip install apache-beam[gcp]. Path specified: gcs://blah
```
Raises immediately because `GCSFileSystem` isn't registered as a subclass.

### Proposed Behavior

Both should behave consistently. GCSFileSystem should be returned from `get_filesystem()` like S3FileSystem, allowing callers to validate dependencies when the filesystem is actually used rather than at lookup time.

### Why This Matters

- Inconsistent API behavior is confusing
- Code that handles multiple filesystem types can't catch/handle GCS gracefully
- Dependency validation at usage time (not lookup time) allows for better error handling and lazy loading patterns

### Environment

- Apache Beam version: 2.70.0
- Python version: 3.11

---

*Generated by Claude Code, confirmed by @hjtran*

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.