Pluggable Authentication Providers for REST Namespace

Summary

Track the implementation of pluggable authentication providers for the Lance REST Namespace, starting with AWS SigV4. The REST Namespace today only supports OAuth2 / Bearer / API Key, which blocks integration with cloud-managed catalogs that use native IAM auth (AWS S3 Tables, API Gateway + IAM Auth). This epic introduces a RestAuthProvider trait in Rust with config-driven selection from Python/Java, and ships SigV4 as the first concrete provider.

Originating proposal: #6583 — see that issue for the full problem statement and prior discussion. Happy to attach a longer design write-up as a comment if reviewers want more depth.

Scope

In scope (this epic)

• RestAuthProvider trait + RestClient integration pipeline
• AWS SigV4 provider (with optional inner-auth delegation)
• Python and Java SDK property pass-through (rest.auth.*)
• Documentation, migration guide, and OpenAPI spec extension

Out of scope (deferred — open separate issues if user demand surfaces)

• OAuth2 auto-refresh provider — the current header.Authorization static-token path covers most users
• Google Cloud / Azure AD providers — open dedicated issues when concrete user requests arrive
• Server-driven auth configuration endpoint (analogous to Iceberg's initSession) — backwards-compatible to add later
• Cross-namespace token cache sharing

Key design decisions

1. No cross-FFI trait objects. Custom auth from Python/Java goes through the existing DynamicContextProvider mechanism. The built-in SigV4 provider is Rust-only and selected by configuration properties.
2. Only one concrete provider in scope: SigV4. NoopAuthProvider is the trivial built-in for the unauthenticated case; Bearer/OAuth2/GCP/Azure are deferred (see Out of scope above). Existing header.Authorization continues to work unchanged.
3. No decorator/delegate composition. Iceberg's RESTSigV4AuthManager wraps OAuth2 by default, but with OAuth2 out of scope here, decorator scaffolding has nothing to wrap. Will be revisited when a second provider lands.
4. Reuse the existing RestClient.apply_headers pipeline. Header application order is base_headers → RestAuthProvider → DynamicContextProvider (the last layer can override auth as an intentional escape hatch).
5. RestNamespaceBuilder::build() returns Result<RestNamespace> instead of RestNamespace, but stays synchronous. Auth initialization (token fetch, AWS credential chain resolution) happens lazily inside each provider on first request — not at framework level, and not in build(). An opt-in warm_up_auth().await provides fail-fast credential validation for callers who prefer eager initialization.

Implementation plan

All work happens in lance-format/lance. Tasks are grouped by category; sequencing is implicit in inter-group dependency. PR-level packaging is left to the implementer.

Core framework

• Define RestAuthProvider trait, RequestContext, and NoopAuthProvider in rust/lance-namespace-impls/src/rest_auth.rs
• Add auth_provider field to RestClient; make apply_headers async and insert auth.authenticate() between base_headers and DynamicContextProvider
• Update RestNamespaceBuilder::from_properties to preserve rest.auth.* keys
• Add RestNamespaceBuilder::auth_provider() setter; change build() to return Result<RestNamespace> (still synchronous)
• Compatibility tests: existing header.Authorization unchanged; rest.auth.type=none identical to no config

AWS SigV4 provider

Once this group lands, all three languages gain SigV4 simultaneously — the Python and Java bindings already forward arbitrary properties to RestNamespaceBuilder::from_properties unchanged.

• Add rest-auth-sigv4 Cargo feature with aws-sigv4, aws-credential-types, aws-smithy-runtime-api (aws-config already available via credential-vendor-aws)
• Implement SigV4AuthProvider in rest_auth/sigv4.rs: parse rest.auth.sigv4.{region,service}, resolve credentials via the AWS chain, sign outgoing requests with the aws-sigv4 crate
• Add create_auth_provider(properties) factory dispatching on rest.auth.type; wire into ConnectBuilder::connect()
• Add RestNamespace::warm_up_auth() for opt-in fail-fast credential validation; called by ConnectBuilder::connect()
• Unit tests for SigV4AuthProvider: property parsing, header generation with deterministic timestamp, error paths (invalid region / missing credentials)

Multi-language regression coverage

Executable documentation and CI safety nets; no new capabilities.

• Python integration test: lance.connect("rest", properties={"rest.auth.type": "sigv4", ...}) against a mock or real endpoint
• Java integration test mirroring the Python one
• List the new rest.auth.* keys in PyRestNamespace::__init__ docstring and Java connect Javadoc

Documentation

• Add an "Authentication" section to REST Namespace docs: SigV4 setup (Rust/Python/Java examples), property reference, AWS credential chain notes, migration from header.Authorization

cc @jackye1995

Start by reading proposal #6583 and the existing RestClient.apply_headers pipeline, then review the task list for rust/lance-namespace-impls/src/rest_auth.rs and rest_auth/sigv4.rs. Run the existing REST namespace tests before adding focused compatibility and SigV4 tests. Done means the provider framework, AWS SigV4 path, Python and Java coverage, and authentication documentation in the listed scope are complete.