Synchronize the storage proxy's configuration file with the manager's DB settings

## Overview

### System Architecture

Backend.AI consists of multiple components that work together:

```
┌─────────────────────────────────────────────────────────────────┐
│ External Artifact Registries │
│ - HuggingFace Hub (models, datasets) │
│ - Reservoir (enterprise artifact registry) │
└─────────────────────────────────────────────────────────────────┘
↓ ↑
(API calls)
↓ ↑
┌─────────────────────────────────────────────────────────────────┐
│ Storage Proxy │
│ - Reads artifact registry configs from TOML file │
│ - Downloads artifacts from external registries │
│ - Manages storage backends (Object Storage, VFS) │
│ - Handles file operations (upload, download, import) │
└─────────────────────────────────────────────────────────────────┘
↑
(HTTP API)
│
┌─────────────────────────────────────────────────────────────────┐
│ Manager │
│ - Stores artifact registry configs in PostgreSQL database │
│ - Stores storage backend configs in PostgreSQL database │
│ - Orchestrates artifact scanning and importing │
│ - Exposes GraphQL/REST APIs to clients │
└─────────────────────────────────────────────────────────────────┘
↑
(GraphQL/REST)
│
┌─────────────────────────────────────────────────────────────────┐
│ Client (Web UI, CLI, API) │
└─────────────────────────────────────────────────────────────────┘
```

### How Configurations Are Used

#### Artifact Registry Configurations (HuggingFace, Reservoir)

- **Manager**: Stores registry metadata (endpoint, credentials) in database tables
- Used for: API responses, UI display, orchestration logic
- **Storage Proxy**: Reads registry configs from TOML file
- Used for: Direct communication with external registries to download artifacts

#### Storage Backend Configurations (Object Storage, VFS)

- **Manager**: Stores storage backend settings in database tables
- Used for: Storage selection, quota management, API responses
- **Storage Proxy**: Reads storage configs from TOML file
- Used for: Actual file I/O operations with storage backends

### Artifact Import Flow Example

1. User requests to import a HuggingFace model via Manager's GraphQL API
1. Manager looks up HuggingFace registry config from `huggingface_registries` table
1. Manager delegates import request to Storage Proxy via HTTP API
1. Storage Proxy looks up HuggingFace config from its TOML file
1. Storage Proxy connects to HuggingFace Hub using TOML credentials
1. Storage Proxy downloads model files and stores them in configured storage backend
1. Storage Proxy looks up storage backend config from TOML to write files

### Why This Duplication Exists

The duplication was introduced because:

- Storage Proxy runs as a separate process/node without database access
- Storage Proxy needs credentials to communicate with external registries
- Storage Proxy needs storage backend settings for file operations
- Initially, TOML configuration was the simplest solution for Storage Proxy

However, this creates maintenance and consistency issues:

- Configuration must be updated in two places (DB + TOML)
- Risk of configuration drift between Manager and Storage Proxy
- No single source of truth for registry/storage settings

----

## Problem Statement

The current artifact registry and storage configurations are duplicated in both the manager's database and the storage proxy's TOML configuration file. This duplication was added for convenience because the storage proxy node cannot access the database.

Let's remove the redundant configuration and modify the system so that the database contents are reflected in the storage proxy.

## 1. Reservoir Registry Configuration Duplication

### Manager Database (`reservoir_registries` table)

Location: `src/ai/backend/manager/models/reservoir_registry.py:34`

The `ReservoirRegistryRow` model contains:

- `id` (UUID): Unique identifier
- `endpoint` (String): Reservoir endpoint URL
- `access_key` (String): Access key for authentication
- `secret_key` (String): Secret key for authentication
- `api_version` (String): API version

### Storage Proxy TOML Configuration

Location: `src/ai/backend/storage/config/unified.py:821` (class `ReservoirConfig`)
Configuration file: `configs/storage-proxy/sample.toml` (`[artifact-registries.*.reservoir]` section)

The TOML configuration contains overlapping fields:

- `endpoint`: Reservoir registry API endpoint
- `manager_endpoint`: Manager API endpoint (for VFS storage)
- `manager_access_key`: Manager access key (for VFS storage)
- `manager_secret_key`: Manager secret key (for VFS storage)
- `manager_api_version`: Manager API version (for VFS storage)
- `object_storage_access_key`: Object storage access key
- `object_storage_secret_key`: Object storage secret key
- `object_storage_region`: Object storage region
- `storage_name`: Reference to storage configuration

### Overlap Mapping

|DB Field|TOML Field (VFS)|TOML Field (Object Storage)|
|---|---|---|
|`endpoint`|`manager_endpoint`|`endpoint`|
|`access_key`|`manager_access_key`|`object_storage_access_key`|
|`secret_key`|`manager_secret_key`|`object_storage_secret_key`|
|`api_version`|`manager_api_version`|N/A|

## 2. HuggingFace Registry Configuration Duplication

### Manager Database (`huggingface_registries` table)

Location: `src/ai/backend/manager/models/huggingface_registry.py:34`

The `HuggingFaceRegistryRow` model contains:

- `id` (UUID): Unique identifier
- `url` (String): HuggingFace endpoint URL
- `token` (String, nullable): Authentication token

### Storage Proxy TOML Configuration

Location: `src/ai/backend/storage/config/unified.py:782` (class `HuggingfaceConfig`)
Configuration file: `configs/storage-proxy/sample.toml` (`[artifact-registries.*.huggingface]` section)

The TOML configuration contains overlapping fields:

- `endpoint`: HuggingFace API endpoint (default: "https://huggingface.co")
- `token`: HuggingFace API token
- `download_chunk_size`: Chunk size for downloads (not in DB)

### Overlap Mapping

|DB Field|TOML Field|
|---|---|
|`url`|`endpoint`|
|`token`|`token`|

## 3. Object Storage Configuration Duplication

### Manager Database (`object_storages` table)

Location: `src/ai/backend/manager/models/object_storage.py:32`

The `ObjectStorageRow` model contains:

- `id` (UUID): Unique identifier
- `name` (String): Storage name
- `host` (String): Storage host
- `access_key` (String): Access key
- `secret_key` (String): Secret key
- `endpoint` (String): Storage endpoint URL
- `region` (String, nullable): Storage region

### Storage Proxy TOML Configuration

Location: `src/ai/backend/storage/config/unified.py:681` (class `ObjectStorageConfig`)
Configuration file: `configs/storage-proxy/sample.toml` (`[[storages]]` section - deprecated, or `[artifact-storages.*.object-storage]` section)

The TOML configuration contains overlapping fields:

- `name`: Storage name
- `endpoint`: Storage endpoint URL
- `access_key`: Access key
- `secret_key`: Secret key
- `region`: Storage region
- `buckets`: List of bucket names (not in DB)
- `presigned_upload`: Presigned upload configuration (not in DB)
- `presigned_download`: Presigned download configuration (not in DB)
- `upload_chunk_size`: Upload chunk size (not in DB)
- `download_chunk_size`: Download chunk size (not in DB)
- `reservoir_download_chunk_size`: Reservoir download chunk size (not in DB)

### Overlap Mapping

|DB Field|TOML Field|
|---|---|
|`name`|`name`|
|`endpoint`|`endpoint`|
|`access_key`|`access_key`|
|`secret_key`|`secret_key`|
|`region`|`region`|
|`host`|(not in TOML)|

## 4. VFS Storage Configuration Duplication

### Manager Database (`vfs_storages` table)

Location: `src/ai/backend/manager/models/vfs_storage.py:27`

The `VFSStorageRow` model contains:

- `id` (UUID): Unique identifier
- `name` (String): Storage name
- `host` (String): Storage host
- `base_path` (String): Base filesystem path

### Storage Proxy TOML Configuration

Location: `src/ai/backend/storage/config/unified.py:628` (class `VFSStorageConfig`)
Configuration file: `configs/storage-proxy/sample.toml` (`[artifact-storages.*.vfs_storage]` section)

The TOML configuration contains overlapping fields:

- `base_path`: Base filesystem path
- `subpath`: Optional subdirectory (not in DB)
- `upload_chunk_size`: Upload chunk size (not in DB)
- `download_chunk_size`: Download chunk size (not in DB)
- `max_file_size`: Maximum file size (not in DB)

### Overlap Mapping

|DB Field|TOML Field|
|---|---|
|`name`|(part of artifact-storages key)|
|`base_path`|`base_path`|
|`host`|(not in TOML)|

## Related Code

### Database Models

- Reservoir Registry: `src/ai/backend/manager/models/reservoir_registry.py:34`
- HuggingFace Registry: `src/ai/backend/manager/models/huggingface_registry.py:34`
- Object Storage: `src/ai/backend/manager/models/object_storage.py:32`
- VFS Storage: `src/ai/backend/manager/models/vfs_storage.py:27`
- Artifact Registries (meta): `src/ai/backend/manager/models/artifact_registries.py:35`

### Config Schemas

- Reservoir: `src/ai/backend/storage/config/unified.py:821` (ReservoirConfig)
- HuggingFace: `src/ai/backend/storage/config/unified.py:782` (HuggingfaceConfig)
- Object Storage: `src/ai/backend/storage/config/unified.py:681` (ObjectStorageConfig)
- VFS Storage: `src/ai/backend/storage/config/unified.py:628` (VFSStorageConfig)

### Migrations

- Reservoir: `src/ai/backend/manager/models/alembic/versions/11ee5c074009_add_reservoir_registries_table.py`
- HuggingFace: `src/ai/backend/manager/models/alembic/versions/2bb529fb88b6_add_huggingface_registries_table.py`

## Note

There's a TODO comment in the codebase (`src/ai/backend/storage/config/unified.py:680`) indicating that object storage configuration should be migrated to the database. This epic addresses that migration for all duplicated configurations.

JIRA Issue: BA-2224