[Python] Bindings for GcsFileSystem FromServiceAccountCredentials
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the enhancement requested
`GcsOptions::FromServiceAccountCredentials` (`cpp/src/arrow/filesystem/gcsfs.h:146`) is the
one credential factory on `GcsOptions` with no Python binding, so a caller holding
[aip/4112] service-account JSON has no way to construct a `GcsFileSystem` from it.
`python/pyarrow/includes/libarrow_fs.pxd` declares the other four (`Defaults`, `Anonymous`,
`FromAccessToken`, `FromImpersonatedServiceAccount`, lines 229-240), and the
credential-routing chain inside `GcsFileSystem.__init__` (`python/pyarrow/_gcsfs.pyx:103-131`,
method at line 88) selects among them. No constructor keyword reaches the fifth. Line
references are against `d048f71`; behaviour checked on pyarrow 25.0.1.
The R bindings already have it. `GcsFileSystem$create()` takes `json_credentials`, either the
JSON itself or a path to a file holding it (`r/R/filesystem.R:611-674`), and routes it to
`GcsOptions::FromServiceAccountCredentials` (`r/src/filesystem.cpp:415-417`); it also rejects
`json_credentials` alongside `access_token`, which is the mutual exclusion proposed below. That
landed in #34524 (GH-34421) in March 2023. So the two bindings have complementary holes: R has
service-account JSON and lacks impersonation (#32207), Python has impersonation
(`target_service_account`) and lacks service-account JSON.
Two routes are open today to an application holding that JSON in memory, and each gives
something up:
- Write it to a file and point `GOOGLE_APPLICATION_CREDENTIALS` at it. That puts the private
key on disk, and it configures the process rather than the filesystem object, so two
filesystems in one process cannot use different service accounts. The docstring on this
function names the case it exists for: "Using this function may be useful when the json
object is obtained from a Cloud Secret Manager or a similar service."
- Mint an access token out of band and pass `access_token` + `credential_token_expiration`.
Per `FromAccessToken`'s docstring those tokens are time limited and the caller refreshes
them manually, and the Python API offers no way to replace the token on an existing
`GcsFileSystem`, so a refresh means constructing a new one.
Concrete case: omniload (panodata/omniload#233) accepts GCS service-account JSON as either a
path or a base64 blob from a secret store. Its S3 and R2 transports moved to `pyarrow.fs` for
the per-file open latency (on that sibling transport, 8.68 ms/file through s3fs against
3.49 ms/file through Arrow, on the reader path that opens one file object per file). GCS
stayed on gcsfs, because the swap would drop that credential mode.
Proposed shape, following the existing chain:
- Declare `FromServiceAccountCredentials` in `libarrow_fs.pxd` beside the other four.
- Add a `GcsFileSystem.__init__` keyword taking the aip/4112 JSON as a string, in the same
base-credential branch as `anonymous` and `access_token` and mutually exclusive with both.
`service_account_json` reads more clearly than `service_account_credentials`, since the
value is serialized JSON rather than a credentials object; the C++ accessor calls it
`json_credentials`. It would stay composable with `target_service_account` by passing the
resulting credentials to `FromImpersonatedServiceAccount`, as the constructor already does
for the other base credentials.
- Pickling would need a policy decision, and I have not assumed one. `__reduce__` round-trips
`access_token` today (`_gcsfs.pyx:183`), but a long-lived private key is a different
proposition and declining to serialize this mode is a reasonable answer. If it should
round-trip through the existing `__reduce__` design, the pxd would also need
`GcsCredentials::json_credentials()` (`gcsfs.h:47`), which is not declared there today but is
already read by the R glue for its own options round-trip (`r/src/filesystem.cpp:489-490`).
Related: #34421 and #34524 (the R binding and its path-or-string handling), #33106 (its
docs), #32207 (the reverse gap, impersonation missing in R), #12763 (the original
`[Python][C++] GCS Bindings`, described there as "mostly based on AWS bindings"; I did not find
service-account JSON discussed on it), #11945 (the C++ credential types this builds on).
Does a constructor keyword fit how you would want this exposed, or would you rather it
arrive as a separate classmethod?
### Component(s)
Python
[aip/4112]: https://google.aip.dev/auth/4112
Contributor guide
Research direction
Start with cpp/src/arrow/filesystem/gcsfs.h:146, python/pyarrow/includes/libarrow_fs.pxd:229-240, and python/pyarrow/_gcsfs.pyx:88-131, then compare the R handling in r/R/filesystem.R:611-674 and r/src/filesystem.cpp:415-417. Done means the Python constructor can route service-account JSON, enforce credential exclusivity, support the existing impersonation path, and settle the __reduce__ policy described in the issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- backend, cloud
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100