[Improvement] Decouple runtime, cloud-storage, and dev dependencies in client-python
- Dominant language
- Java
- Stars
- 3.2k
- Forks
- 935
- Avg merge
- 1d 17h
- Merged PRs (30d)
- 339
Description
### What would you like to be improved?
`pip install apache-gravitino` (verified against PyPI metadata for v1.3.0) forces installation of 13 exact-pinned packages as unconditional runtime dependencies. Several of these are development tools that are never imported at runtime:
| Package | Actual usage | Runtime import? |
|---------|-------------|-----------------|
| `black==26.3.1` | Code formatter | No |
| `flake8==7.0.0` | Linter | No |
| `pre-commit==3.5.0` | Git hooks | No |
| `requests==2.32.5` | Integration tests only | No |
The remaining packages (`pyarrow`, `s3fs`, `gcsfs`, `ossfs`, `adlfs`) are used at runtime but only through lazy `importlib.import_module()` calls in specific storage handlers (`gvfs_storage_handler.py`). They are optional by nature — a user accessing only S3 never triggers the GCS/Azure/OSS/HDFS code paths — yet all are installed unconditionally.
The exact `==` pinning also causes frequent pip resolution conflicts when users have other packages requiring different versions of shared transitive deps like `fsspec`.
For comparison, SDKs like `databricks-sdk` use flexible version ranges (`requests<3,>=2.28.1`) and only declare true runtime deps, with optional backends under extras.
### How should we improve?
Separate dependencies into core runtime + optional extras:
```
# Core runtime (always installed; use CI-validated lower bounds,
# add upper bounds only where known incompatibilities exist)
dataclasses-json>=0.6.7
readerwriterlock>=1.0.9
fsspec>=2024.1.0
cachetools>=7.0
# Optional storage extras (user opts in per provider)
[s3] → s3fs (version-compatible with fsspec floor)
[gcs] → gcsfs
[oss] → ossfs
[azure] → adlfs
[hdfs] → pyarrow
[cloud] → s3fs, gcsfs, ossfs, adlfs (all cloud backends, no HDFS)
# Dev/test (CI and contributors)
[test] → black, flake8, pre-commit, pytest, requests, pyarrow,
s3fs, gcsfs, ossfs, adlfs, pandas, coverage, ...
```
Note: The project already has a `[lance]` extra. The aggregation extra is named `[cloud]` rather than `[all]` to avoid ambiguity about whether it includes `[hdfs]` and `[lance]`.
**Breaking change mitigation:**
Removing cloud backends from the default install is a user-visible breaking change — existing users who run `pip install apache-gravitino` and then access S3/GCS/GVFS will get a runtime `ModuleNotFoundError`. To mitigate:
1. Each storage handler's `get_filesystem()` should catch `ModuleNotFoundError` and raise a descriptive error with install guidance, e.g.: `Install "apache-gravitino[s3]" to use s3a:// locations.`
2. Target this change for the next major release (2.0.0), or introduce a two-phase migration:
- Phase 1 (minor release): Add extras, emit a deprecation warning when a lazily-imported storage backend is resolved from the default install rather than an explicit extra.
- Phase 2 (major release): Remove cloud/hdfs packages from core `install_requires`.
**Version constraints:** Use CI-validated lower bounds for each runtime dep. Add upper bounds only where known incompatibilities exist. The `fsspec` ecosystem packages (`gcsfs`, `s3fs`, `ossfs`, `adlfs`) have tight cross-version coupling and should be validated via an installation matrix in CI, not just by sharing a single floor version.
**Test environment:** Gradle currently installs `.[dev]` (`build.gradle.kts:161`). After this change, CI/Gradle should install `.[test]` which aggregates dev tools + all storage backends + HDFS, ensuring no test breakage.
**Relationship with #7779:** This is complementary to the `uv` migration. #7779 addresses build-side tooling (Gradle + Miniforge → uv); this issue addresses the user-facing install contract. PR #7811 could incorporate this layering in `pyproject.toml`'s `[project.dependencies]` + `[project.optional-dependencies]`, or it can be implemented independently.
Related: #7779, #7811
Contributor guide
Research direction
Start with pyproject.toml and build.gradle.kts:161 to inspect the current dependency groups and CI installation path. Then read gvfs_storage_handler.py and its get_filesystem() entry points, along with the relevant storage tests. Done means core installs omit optional backends and development tools, extras remain CI-covered, and missing backends provide installation guidance.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, build-system, ci-cd
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100