Apache Iceberg — Catalog management, data ingestion, and table maintenance tasks

## Summary

The `plugin-iceberg` plugin enables Kestra to manage Apache Iceberg table catalogs, ingest data directly from internal Kestra storage (CSV, JSON, Parquet) into Iceberg tables, and automate data lake maintenance operations (compaction, snapshot expiry, orphan file cleanup) — all without requiring Spark or any compute cluster at runtime.

## Motivation

Data engineering teams using Iceberg on AWS Glue, Nessie, REST catalogs, or Hive Metastore today must orchestrate schema changes, data ingestion, and maintenance jobs via Spark or heavy compute frameworks. Kestra can own this control plane natively: lightweight Java clients for each catalog type let users manage namespaces and tables, write data, and run governance tasks directly from a flow — removing the overhead of spinning up Spark clusters for administrative operations. Primary beneficiaries are data platform engineers and lake house teams who want reliable, auditable orchestration over their Iceberg catalogs without the complexity of Spark session management.

## Context

Related closed placeholder: https://github.com/kestra-io/kestra/issues/1450 (original Iceberg tracking issue in the core repo).

Reference implementations to model after:
- `plugin-jdbc` — connection/catalog abstraction patterns and abstract base task design
- `plugin-aws` — AWS-specific credential configuration and Glue catalog integration patterns

## API Reference

- **Official docs**: https://iceberg.apache.org/docs/latest/java-api-quickstart/
- **Authentication**: Catalog-specific — REST catalogs use bearer tokens or OAuth2; AWS Glue uses IAM credentials; Hive Metastore uses Kerberos or plaintext; Nessie uses bearer tokens. All authentication properties are passed through the `catalogConfig: Map` property block using Iceberg's native property keys.
- **Base URL pattern**: REST catalog: `http(s):///v1/`; others configured via catalog-specific property keys (e.g. `glue.endpoint`, `hive.metastore.uris`, `nessie.uri`)
- **SDK / client library**: `org.apache.iceberg:iceberg-api` + catalog-specific sub-modules (see Gradle Dependencies below)

## Gradle Dependencies

Add to `build.gradle`:

```groovy
// Apache Iceberg core API and REST catalog
implementation "org.apache.iceberg:iceberg-api:1.6.1"
implementation "org.apache.iceberg:iceberg-core:1.6.1"

// Parquet support for AppendData (no Spark dependency)
implementation "org.apache.iceberg:iceberg-parquet:1.6.1"
implementation "org.apache.parquet:parquet-hadoop:1.14.1"

// AWS Glue + S3 catalog
implementation "org.apache.iceberg:iceberg-aws:1.6.1"

// Nessie catalog
implementation "org.apache.iceberg:iceberg-nessie:1.6.1"

// Hive Metastore catalog
implementation "org.apache.iceberg:iceberg-hive-metastore:1.6.1"

// Integration tests only — tabulario/spark-iceberg container as REST catalog fixture
testImplementation "org.testcontainers:testcontainers:1.19.8"
```

> Use the latest stable version available on Maven Central.
> **Critical constraint**: Do NOT include Spark or Hadoop runtime dependencies in `implementation` scope. Any Hadoop filesystem adapter must be scoped to `testImplementation` only.

## Plugin Structure

- **Repository**: `plugin-iceberg`
- **Namespace**: `io.kestra.plugin.iceberg`
- **Sub-plugins**: `catalog` (control-plane tasks), `data` (ingestion tasks), `maintenance` (compaction and cleanup)
- **Categories**: `DATA`

## Suggested Tasks

### Base Connectivity
Implement `AbstractIcebergConnection` — abstract base task holding a `catalogConfig: Map` property. Use `CatalogUtil.buildIcebergCatalog(name, properties, hadoopConf)` to instantiate the correct catalog at runtime. Supported catalog types via property keys: `rest`, `glue`, `hive`, `nessie`.

### Control Plane (`catalog` sub-package)
1. `CreateNamespace` — create a namespace with optional properties map
2. `DeleteNamespace` — drop a namespace (with `ifExists` flag)
3. `CreateTable` — create a new Iceberg table with a declarative schema definition
4. `AlterTable` — declarative schema evolution: add/rename/drop columns, update column types, set/unset table properties

### Data Plane (`data` sub-package)
5. `AppendData` — read a Kestra internal storage file (CSV / JSON / Parquet) and append rows to an Iceberg table using the Iceberg `AppendFiles` API directly (no Spark required)

### Maintenance (`maintenance` sub-package)
6. `RewriteFiles` — compact small data files into optimally-sized Parquet files via `Actions.forTable().rewriteDataFiles().execute()`
7. `ExpireSnapshots` — expire snapshots older than a configurable retention window (`olderThanDays` or absolute timestamp)
8. `RemoveOrphanFiles` — identify and delete data files not referenced by any live snapshot

### Testing & Documentation
9. Provide `docker-compose-ci.yml` using the `tabulario/spark-iceberg` image as an ephemeral local REST catalog + MinIO object storage fixture for integration tests (CI and local dev only — not a runtime dependency)
10. Write unit + integration tests connecting to the REST catalog endpoint exposed by the test container
11. Add `package-info.java` per sub-package with `@PluginSubGroup(category = PluginSubGroup.PluginCategory.DATA)`
12. Add `metadata/index.yaml` and plugin icon SVG
13. Add YAML examples and plugin documentation

## YAML Examples

### Example 1 — Create a namespace and register a table schema

```yaml
id: iceberg_setup
namespace: company.team

tasks:
- id: create_namespace
type: io.kestra.plugin.iceberg.catalog.CreateNamespace
catalogConfig:
type: rest
uri: "{{ secret('ICEBERG_REST_URI') }}"
credential: "{{ secret('ICEBERG_TOKEN') }}"
warehouse: s3://my-bucket/warehouse
namespace: analytics

- id: create_table
type: io.kestra.plugin.iceberg.catalog.CreateTable
catalogConfig:
type: rest
uri: "{{ secret('ICEBERG_REST_URI') }}"
credential: "{{ secret('ICEBERG_TOKEN') }}"
warehouse: s3://my-bucket/warehouse
namespace: analytics
tableName: events
schema:
fields:
- name: event_id
type: string
required: true
- name: ts
type: timestamptz
required: true
- name: payload
type: string
```

### Example 2 — Append a CSV file from internal storage to an Iceberg table (AWS Glue)

```yaml
id: iceberg_ingest
namespace: company.team

tasks:
- id: fetch_data
type: io.kestra.plugin.core.http.Download
uri: "https://data.example.com/events.csv"

- id: append_to_iceberg
type: io.kestra.plugin.iceberg.data.AppendData
catalogConfig:
type: glue
warehouse: "s3://my-bucket/warehouse"
glue.id: "{{ secret('AWS_ACCOUNT_ID') }}"
namespace: analytics
tableName: events
from: "{{ outputs.fetch_data.uri }}"
format: CSV
```

### Example 3 — Nightly maintenance: expire snapshots and compact data files

```yaml
id: iceberg_maintenance
namespace: company.team

triggers:
- id: nightly
type: io.kestra.plugin.core.trigger.Schedule
cron: "0 2 * * *"

tasks:
- id: expire_snapshots
type: io.kestra.plugin.iceberg.maintenance.ExpireSnapshots
catalogConfig:
type: rest
uri: "{{ secret('ICEBERG_REST_URI') }}"
credential: "{{ secret('ICEBERG_TOKEN') }}"
warehouse: s3://my-bucket/warehouse
namespace: analytics
tableName: events
olderThanDays: 7

- id: compact_files
type: io.kestra.plugin.iceberg.maintenance.RewriteFiles
catalogConfig:
type: rest
uri: "{{ secret('ICEBERG_REST_URI') }}"
credential: "{{ secret('ICEBERG_TOKEN') }}"
warehouse: s3://my-bucket/warehouse
namespace: analytics
tableName: events
targetFileSizeBytes: 134217728
```

## Acceptance Criteria

- [ ] `AbstractIcebergConnection` with `catalogConfig: Map` instantiates the correct Iceberg catalog for REST, AWS Glue, Hive Metastore, and Nessie
- [ ] Control-plane tasks implemented: `CreateNamespace`, `DeleteNamespace`, `CreateTable`, `AlterTable`
- [ ] Data-plane task `AppendData` reads CSV / JSON / Parquet from Kestra internal storage and appends to an Iceberg table without any Spark dependency
- [ ] Maintenance tasks implemented: `RewriteFiles`, `ExpireSnapshots`, `RemoveOrphanFiles`
- [ ] `docker-compose-ci.yml` uses `tabulario/spark-iceberg` strictly as an integration-test fixture — not referenced anywhere in the plugin's production build
- [ ] Plugin production build contains **no Spark or Docker runtime dependencies** (verify with `./gradlew dependencies`)
- [ ] All `Property` fields support Kestra expression language (template rendering)
- [ ] Unit + integration tests pass (`./gradlew test`)
- [ ] `package-info.java` per sub-package with `@PluginSubGroup(category = PluginSubGroup.PluginCategory.DATA)`
- [ ] `metadata/index.yaml` and plugin icon SVG present
- [ ] Build passes with `./gradlew build`

## Repository Setup Checklist

### 1. Add to Sanity check page

Add this plugin to the [Sanity check Notion page](https://www.notion.so/kestra-io/32736907f7b580cbb00dc7c061e624b1?v=32736907f7b58002ac2b000ccc63d8a2).

### 2. Run scoped Terraform apply

Run the following from `infra/terraform/github`:

```bash
terraform apply \
-target='github_repository.repo["plugin-iceberg"]' \
-target='github_issue_labels.plugins["plugin-iceberg"]' \
-target='github_repository_ruleset.branch["plugin-iceberg"]'
```