ingests parquet into redshift via ADBC

# Proposal: ADBC Support for Redshift Destination

## Overview

This proposal outlines adding ADBC (Arrow Database Connectivity) support for the Redshift destination, enabling direct parquet loading without requiring users to configure a separate staging destination in dlt.

## Background

### Current Redshift Loading Methods

Currently, dlt supports two loading methods for Redshift:

1. **`insert_values`** - Direct SQL INSERT statements (slow for large datasets)
2. **Staging via S3** - User configures S3 as a staging destination, dlt uploads files to S3, then Redshift COPYs from S3

The staging approach requires users to:
- Configure a separate staging destination (filesystem with S3)
- Manage S3 bucket permissions and IAM roles
- Understand dlt's staging concept

### ADBC Approach

The [ADBC Redshift driver](https://github.com/columnar-tech/adbc-quickstarts/tree/main/python/redshift) provides a third option that simplifies the user experience while still leveraging S3 for optimal bulk loading performance.

## How ADBC Redshift Works

### Architecture

```
┌─────────────────────────────────────────────────────────────────┐
│ Current dlt Staging │
├─────────────────────────────────────────────────────────────────┤
│ Extract → Normalize → [Parquet Files] │
│ ↓ │
│ dlt uploads to S3 │
│ ↓ │
│ dlt creates reference job │
│ ↓ │
│ dlt executes COPY FROM S3 │
│ ↓ │
│ (user manages S3 cleanup) │
└─────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────┐
│ ADBC Approach │
├─────────────────────────────────────────────────────────────────┤
│ Extract → Normalize → [Parquet Files] │
│ ↓ │
│ Read as Arrow batches │
│ ↓ │
│ ADBC driver.adbc_ingest() handles: │
│ - Upload to S3 (temporary) │
│ - Execute COPY FROM S3 │
│ - Cleanup S3 files │
└─────────────────────────────────────────────────────────────────┘
```

### Why S3 is Still Required

Redshift's architecture requires S3 for efficient bulk loading - there is no direct "insert from Arrow" path. The ADBC driver abstracts this by:

1. Taking Arrow record batches as input
2. Writing them to a configured S3 bucket (`redshift.ingest.bucket`)
3. Executing the COPY command
4. Cleaning up temporary S3 files

This is documented in the ADBC driver error message:
```
INVALID_STATE: [redshift] Must set redshift.ingest.bucket to ingest data
```

**Key Documentation:**
- [ADBC Redshift Quickstart](https://github.com/columnar-tech/adbc-quickstarts/tree/main/python/redshift#readme) - Shows connection configuration including `redshift.ingest.bucket`
- [Columnar Tech dbc CLI](https://columnar.tech/dbc/) - Tool for installing ADBC drivers
- [Apache Arrow ADBC Specification](https://arrow.apache.org/adbc/) - ADBC API standard

## Benefits Over Current Staging

| Feature | Current Staging | ADBC |
|---------|-----------------|------|
| User configuration | Staging destination + S3 creds | S3 bucket in destination config |
| S3 upload | dlt handles | Driver handles |
| COPY execution | dlt generates SQL | Driver handles |
| S3 cleanup | User/dlt responsibility | Driver handles automatically |
| Data path | File → S3 → Redshift | Arrow batches → S3 → Redshift |
| Type fidelity | File format dependent | Arrow-native |

## Proposed Implementation

### 1. Configuration Changes

Add new configuration options to `RedshiftClientConfiguration`:

```python
@configspec
class RedshiftClientConfiguration(PostgresClientConfiguration):
destination_type: Final[str] = "redshift"
credentials: RedshiftCredentials = None

# Existing
staging_iam_role: Optional[str] = None

# New for ADBC
adbc_ingest_bucket: Optional[str] = None # S3 bucket for ADBC ingestion
```

### 2. Factory Changes

Update capabilities to support parquet as a loader format:

```python
def _raw_capabilities(self) -> DestinationCapabilitiesContext:
caps = DestinationCapabilitiesContext()
caps.preferred_loader_file_format = "insert_values"
caps.supported_loader_file_formats = ["insert_values", "parquet", "model"]
caps.loader_file_format_selector = make_adbc_parquet_file_format_selector(
"redshift",
"https://dlthub.com/docs/dlt-ecosystem/destinations/redshift#adbc-loading",
prefer_parquet=False, # Don't prefer by default, requires bucket config
)
# ... rest unchanged
```

### 3. ADBC Job Implementation

```python
class RedshiftParquetCopyJob(AdbcParquetCopyJob):
def _connect(self) -> "Connection":
from adbc_driver_manager import dbapi

self._config = self._job_client.config

db_kwargs = {
"uri": self._config.credentials.to_native_representation(),
}

# Add ingest bucket if configured
if self._config.adbc_ingest_bucket:
db_kwargs["redshift.ingest.bucket"] = self._config.adbc_ingest_bucket

# Add IAM role if configured (for S3 access)
if self._config.staging_iam_role:
db_kwargs["redshift.iam_role"] = self._config.staging_iam_role

return dbapi.connect(driver="redshift", db_kwargs=db_kwargs)
```

### 4. Fallback Behavior

The `loader_file_format_selector` should:
1. Check if ADBC driver is installed (`dbc install redshift`)
2. Check if `adbc_ingest_bucket` is configured
3. Only enable parquet if both conditions are met
4. Fall back to `insert_values` otherwise

## User Experience

### Before (Staging Required)

```python
import dlt

pipeline = dlt.pipeline(
pipeline_name="my_pipeline",
destination="redshift",
staging="filesystem", # Required for parquet
dataset_name="my_data",
)

# Also need to configure in secrets.toml:
# [destination.filesystem]
# bucket_url = "s3://my-bucket"
# [destination.filesystem.credentials]
# aws_access_key_id = "..."
# aws_secret_access_key = "..."
```

### After (ADBC)

```python
import dlt

pipeline = dlt.pipeline(
pipeline_name="my_pipeline",
destination=dlt.destinations.redshift(
adbc_ingest_bucket="my-bucket", # Simple!
),
dataset_name="my_data",
)
```

Or in `secrets.toml`:
```toml
[destination.redshift]
adbc_ingest_bucket = "my-bucket"
staging_iam_role = "arn:aws:iam::123456789:role/RedshiftS3Role"
```

## Dependencies

- `adbc-driver-manager>=1.8.0` (already in `pyproject.toml` under `[project.optional-dependencies.adbc]`)
- Redshift ADBC driver (installed via `dbc install redshift`)

## Documentation Links

- [ADBC Redshift Quickstart](https://github.com/columnar-tech/adbc-quickstarts/tree/main/python/redshift#readme) - Complete example with connection configuration
- [Columnar Tech dbc CLI](https://columnar.tech/dbc/) - Tool for installing and managing ADBC drivers
- [ADBC Driver Manager (PyPI)](https://pypi.org/project/adbc-driver-manager/) - Python ADBC driver manager
- [Apache Arrow ADBC Specification](https://arrow.apache.org/adbc/) - ADBC API standard and documentation
- [Amazon Redshift COPY Command](https://docs.aws.amazon.com/redshift/latest/dg/r_COPY.html) - Redshift COPY documentation

## Testing

Add Redshift to the existing ADBC tests in `tests/load/pipeline/test_adbc_loading.py`:

```python
@pytest.mark.parametrize(
"destination_config",
destinations_configs(
default_sql_configs=True,
subset=["postgres", "mssql", "redshift", "sqlalchemy"]
),
ids=lambda x: x.name,
)
def test_adbc_parquet_loading(destination_config: DestinationTestConfiguration):
# Test requires ADBC driver and ingest bucket to be configured
# Skip if bucket not configured or driver not installed
...
```

## Open Questions

1. **Should we auto-derive the bucket from staging config?** If a user already has S3 staging configured, we could potentially reuse that bucket for ADBC ingestion.

2. **AWS credentials handling** - The ADBC driver may use the default AWS credential chain. Should we support passing explicit credentials?

3. **Cluster type configuration** - The driver supports different cluster types (`redshift-serverless`, `redshift-iam`, `redshift`). Should this be configurable?

4. **Error handling** - How should we handle the case where the bucket is configured but the driver is not installed? Should we provide clear error messages?

## Conclusion

Adding ADBC support for Redshift provides a simpler user experience for bulk loading while maintaining the performance benefits of S3-based COPY operations. The driver handles the S3 lifecycle automatically, reducing configuration complexity and potential for errors. This approach maintains compatibility with Redshift's architecture while abstracting away the complexity of managing S3 staging manually.

Start by reading RedshiftClientConfiguration, the Redshift factory capabilities, and the existing ADBC implementation pattern. Run the ADBC tests in tests/load/pipeline/test_adbc_loading.py, including the Redshift configuration when available. Done means parquet loading is enabled only with the driver and ingest bucket configured, with insert_values remaining the fallback.

Written by the indexing model from the issue text.