google / google/etils

S3 paths fail when TensorFlow is installed due to tf_backend override

Open
#794 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
259
Forks
34
Avg merge
57m
Merged PRs (30d)
1

Description

**Summary**

When TensorFlow is installed, epath.Path routes S3 paths to tf_backend, even though TensorFlow no longer supports the S3 scheme natively.

**The issue**

The _backend property in etils/epath/gpath.py unconditionally overrides the backend for all non-None URI schemes:
```
@property
def _backend(self):
try:
backend = _PREFIX_TO_BACKEND[self._uri_scheme]
if _is_tf_installed() and self._uri_scheme is not None:
return backend_lib.tf_backend # overrides s3 too
return backend
except KeyError:
...
```
Since TensorFlow 2.6, S3 filesystem support was removed from core TensorFlow and moved to the separate tensorflow-io package tensorflow/tensorflow#51583, tensorflow/tensorflow#5359. Core `tf.io.gfile` no longer registers the S3 scheme.

I verified this on TensorFlow 2.19.1:

```
>>> import tensorflow as tf
>>> print('TF version:', tf.__version__)
TF version: 2.19.1
>>> print('Registered schemes:', tf.io.gfile.get_registered_schemes())
Registered schemes: ['gs', 'ram', 'file', '']
>>> print('S3 supported:', 's3' in tf.io.gfile.get_registered_schemes())
S3 supported: False
```
Only `gs` (GCS) remains as a built-in cloud filesystem scheme. S3 is not registered. Since etils does not depend on `tensorflow-io`, routing S3 paths to` tf_backend` will always fail.

**Reproduction**:

```
>>> from etils import epath
>>> epath.Path('s3://my-bucket/my-path').exists()
Traceback (most recent call last):
File "", line 1, in
File "/usr/local/lib/python3.10/dist-packages/etils/epath/gpath.py", line 152, in exists
return self._backend.exists(self._path_str)
File "/usr/local/lib/python3.10/dist-packages/etils/epath/backend.py", line 278, in exists
return self.gfile.exists(path)
File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/lib/io/file_io.py", line 290, in file_exists_v2
_pywrap_file_io.FileExists(compat.path_to_bytes(path))
tensorflow.python.framework.errors_impl.UnimplementedError: File system scheme 's3' not implemented (file: 's3://my-bucket/my-path')
```

Setting `EPATH_USE_TF=false` works around this, but users should not need to know about this env var to use the documented `etils[epath-s3]` extra.

**Request**:

The retro-compatibility concern in the code comment makes sense for `gs://`, where TF still has built-in support. However, since TF 2.6+ no longer registers the S3 scheme, there is no backward compatibility to preserve for it.

Would it be possible to limit the `tf_backend` override to only `gs://`? For example:
```
if _is_tf_installed() and self._uri_scheme == 'gs':
return backend_lib.tf_backend
```
This would preserve the existing behavior for GCS users while allowing S3 to use its registered `fsspec_backend` as intended.

**Related**:

- #168 (same symptom — was closed as a TensorFlow issue, but the root cause is epath's routing logic)
- tensorflow/tensorflow#51583 — S3 support moved out of core TF
- tensorflow/tensorboard#5359 — confirms tf.io.gfile no longer supports S3 without tensorflow-io
- tensorflow/tensorflow#53818 — "File system scheme 's3' not implemented"

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.