Feature Request: Support Per-Base Independent Credentials in Multi-Base Datasets

Feature Request: Support Per-Base Independent Credentials in Multi-Base Datasets

Summary

Currently, when a single Lance Dataset spans multiple storage buckets (via Manifest.base_paths), all base paths share the same ObjectStoreParams (including credentials). This means it is impossible to use different access keys for different buckets within the same Dataset. This feature request proposes adding per-base credential isolation so that each BasePath can optionally carry its own storage_options.

Motivation

In production multi-tenant or cross-account environments, different buckets often belong to different cloud accounts or require different IAM roles. For example:

• Cross-account storage: s3://team-a-bucket/ uses Account A's credentials, while s3://team-b-bucket/ uses Account B's credentials
• Multi-cloud storage: A Dataset has data in both s3://aws-bucket/ (AWS credentials) and cos://tencent-bucket/ (Tencent COS credentials)
• Least-privilege access: Different buckets require different scoped credentials with minimal permissions

Currently, the only workaround is to ensure a single IAM Role has access to all buckets, which violates the principle of least privilege and is not always possible in cross-account scenarios.

Current Behavior (with source code evidence)

1. BasePath has no credential field

// Source: rust/lance-table/src/format/manifest.rs
pub struct BasePath {
    pub id: u32,
    pub name: Option<String>,
    pub is_dataset_root: bool,
    /// The full URI string (e.g., "s3://bucket/path")
    pub path: String,
    // ❌ No storage_options or credentials field
}

2. object_store_for_base() shares a single store_params

// Source: rust/lance/src/dataset.rs
pub(crate) async fn object_store_for_base(&self, base_id: u32) -> Result<Arc<ObjectStore>> {
    let base_path = self.manifest.base_paths.get(&base_id).ok_or_else(|| {
        Error::invalid_input(format!("Dataset base path with ID {} not found", base_id))
    })?;

    let (store, _) = ObjectStore::from_uri_and_params(
        self.session.store_registry(),
        &base_path.path,
        &self.store_params.as_deref().cloned().unwrap_or_default(),
        //  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        //  ⚠️ Same store_params (including credentials) for ALL base paths
    )
    .await?;

    Ok(store)
}

The self.store_params is Dataset-level shared — every base_id uses the same ObjectStoreParams, including the same storage_options_accessor (credentials).

3. Cache key design already supports credential differentiation

// Source: rust/lance-io/src/object_store.rs
pub struct ObjectStoreParams {
    // ...
    pub storage_options_accessor: Option<Arc<StorageOptionsAccessor>>,
    // ...
}

The ObjectStoreRegistry cache key is (prefix, ObjectStoreParams), and ObjectStoreParams's Hash implementation includes the accessor_id() from storage_options_accessor. This means if different ObjectStoreParams are provided, they will be cached as separate ObjectStore instances — the cache layer already supports this.

4. Credential refresh is also Dataset-level shared

When using LanceNamespaceStorageOptionsProvider, the describe_table() call returns one set of credentials that is applied to the entire Dataset. There is no mechanism to return per-bucket credentials.

Proposed Solution

Option A: Add storage_options to BasePath (Recommended)

This is the minimal-invasive approach that preserves backward compatibility.

Step 1: Extend BasePath with an optional storage_options field:

pub struct BasePath {
    pub id: u32,
    pub name: Option<String>,
    pub is_dataset_root: bool,
    pub path: String,
    /// Per-base storage options (credentials, endpoint, etc.)
    /// When None, falls back to the Dataset-level store_params.
    pub storage_options: Option<HashMap<String, String>>,
}

Step 2: Modify object_store_for_base() to construct per-base ObjectStoreParams:

pub(crate) async fn object_store_for_base(&self, base_id: u32) -> Result<Arc<ObjectStore>> {
    let base_path = self.manifest.base_paths.get(&base_id).ok_or_else(|| {
        Error::invalid_input(format!("Dataset base path with ID {} not found", base_id))
    })?;

    let params = if let Some(ref base_options) = base_path.storage_options {
        // Merge base-specific options on top of dataset-level params
        let mut params = self.store_params.as_deref().cloned().unwrap_or_default();
        let accessor = StorageOptionsAccessor::with_static_options(base_options.clone());
        params.storage_options_accessor = Some(Arc::new(accessor));
        params
    } else {
        self.store_params.as_deref().cloned().unwrap_or_default()
    };

    let (store, _) = ObjectStore::from_uri_and_params(
        self.session.store_registry(),
        &base_path.path,
        &params,
    )
    .await?;

    Ok(store)
}

Step 3: Update protobuf serialization for BasePath to include storage_options:

message BasePath {
  uint32 id = 1;
  optional string name = 2;
  bool is_dataset_root = 3;
  string path = 4;
  // New field: per-base storage options
  map<string, string> storage_options = 5;
}

Step 4: Update WriteParams to accept per-base credentials:

// In WriteParams or initial_bases configuration
BasePath {
    id: 2,
    name: Some("cross-account-bucket".to_string()),
    path: "s3://other-account-bucket/tables/t1".to_string(),
    is_dataset_root: true,
    storage_options: Some(HashMap::from([
        ("aws_access_key_id".to_string(), "OTHER_KEY".to_string()),
        ("aws_secret_access_key".to_string(), "OTHER_SECRET".to_string()),
    ])),
}

Option B: Per-Base StorageOptionsProvider

For dynamic credential refresh scenarios, allow each BasePath to have its own StorageOptionsProvider:

pub struct BasePath {
    pub id: u32,
    pub name: Option<String>,
    pub is_dataset_root: bool,
    pub path: String,
    pub storage_options: Option<HashMap<String, String>>,
    // Dynamic provider for credential refresh (runtime only, not serialized)
    #[serde(skip)]
    pub storage_options_provider: Option<Arc<dyn StorageOptionsProvider>>,
}

Scope of Changes

Component	Change Required
lance-table/src/format/manifest.rs	Add storage_options field to BasePath
lance-table/src/format/pb.rs (protobuf)	Update BasePath message definition
lance/src/dataset.rs	Modify object_store_for_base() to use per-base params
lance/src/dataset/write.rs	Accept per-base credentials in WriteParams
lance-namespace/src/namespace.rs	(Optional) Extend DescribeTableResponse to return per-base credentials
Python bindings	Expose per-base storage_options in write_dataset() and BasePath
Java bindings	Expose per-base storage options in Java API

Security Considerations

• Credential serialization: Per-base storage_options containing credentials will be serialized into the Manifest file. This is the same pattern used by the Dataset-level storage_options, but users should be aware that credentials are stored in plaintext in the Manifest. Consider supporting credential references (e.g., secret ARNs) instead of raw credentials.
• Credential refresh: For dynamic credentials (STS tokens), the per-base StorageOptionsProvider approach (Option B) should be preferred over static options, since static tokens in the Manifest will expire.

Backward Compatibility

• The storage_options field in BasePath is optional (Option<HashMap<String, String>>), so existing Manifests without this field will continue to work.
• The protobuf field uses a new field number, so older readers will simply ignore it.
• The fallback behavior (use Dataset-level store_params when base_path.storage_options is None) ensures full backward compatibility.

Related Context

• The ObjectStoreRegistry cache already differentiates by ObjectStoreParams in the cache key, so per-base credentials will naturally produce separate ObjectStore instances.
• The LanceNamespaceStorageOptionsProvider currently returns a single set of credentials via describe_table(). The Namespace API may need to be extended to support per-base credential vending in the future.