airbytehq / airbytehq/PyAirbyte

feat: Support mapper/mapping preservation in cross-workspace connection migration

Offen
#980 1 Kommentar 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
Python
Sterne
344
Forks
77
Ø Merge
1 T. 11 Std.
Gemergte PRs (30 T.)
35

Beschreibung

## Summary

Ensure that stream mappers (hashing, field-renaming, row-filtering, encryption, field-filtering) are properly handled during cross-workspace connection migration workflows.

**Requested by:** @aaronsteers
**Related Devin session:** https://app.devin.ai/sessions/eb636b8d14fc4895b3b894ea399f998e
**Related issues:**
- https://github.com/airbytehq/PyAirbyte/issues/978 (parent migration issue)
- Cross-linked Ops MCP issue: TBD (will be created and linked)

---

## Current State

### What works today

Mappers **are** preserved during catalog dump/import via the existing tools:

1. **Catalog dump** (`dump_raw_catalog()` → `web_backend/connections/get`): The API response includes `mappers` per stream in the `syncCatalog`, with each mapper having `type` and `mapperConfiguration`.

2. **Catalog import** (`import_raw_catalog()` → `web_backend/connections/update`): The platform's `CatalogConverter.toConfiguredInternal()` correctly converts mappers from API model to internal model (lines 305-312 of `CatalogConverter.kt`), preserving `type` and `mapperConfiguration`.

So the basic mapper round-trip (dump from source connection → import to target connection) **does work** for mappers without secrets.

### What does NOT work

1. **Mapper secrets are masked on dump**: The platform's `MapperSecretHelper.maskMapperSecrets()` masks secret fields (e.g., encryption keys) before returning the catalog via the API. This means:
- When dumping a catalog from the source connection, mapper secrets (e.g., AES encryption keys) will be replaced with `"**********"` or similar masked values
- When importing the dumped catalog to the target connection, the masked values won't work
- The user would need to manually re-enter mapper secrets for the target connection

2. **No MCP tools for mapper management**: Neither Coral MCP nor Ops MCP has any tools for:
- Listing mappers configured on a connection
- Adding/removing/updating individual mappers
- Validating mappers against the target connection's streams (the platform has a `/v1/web_backend/mappers/validate` endpoint, but it's not exposed via MCP)

3. **No mapper compatibility validation**: When migrating, there's no check that the target workspace/platform version supports all the mapper types used by the source connection.

---

## Supported Mapper Types

From the platform's `StreamMapperType` enum:

| Mapper Type | Has Secrets? | Migration Concern |
|---|---|---|
| `hashing` | No | Should migrate cleanly |
| `field-renaming` | No | Should migrate cleanly |
| `row-filtering` | No | Should migrate cleanly |
| `field-filtering` | No | Should migrate cleanly |
| `encryption` | **Yes** (encryption keys) | Secrets will be masked, need re-entry |

---

## Proposed Improvements (P2 - not blocking initial migration)

### 1. Document mapper secret limitation
At minimum, document that mapper secrets (currently only `encryption` mappers) will need to be manually re-entered after migration.

### 2. Add mapper listing to describe_cloud_connection
Enhance the `describe_cloud_connection` MCP tool to include a summary of configured mappers per stream (type + non-secret config), so users can see what needs attention.

### 3. Add mapper validation tool
Expose the platform's `/v1/web_backend/mappers/validate` endpoint as an MCP tool to validate that mappers from a dumped catalog are compatible with the target connection.

---

## Priority

**P2** — Not blocking for initial migration. Basic mapper preservation works via catalog dump/import. Only encryption mapper secrets need manual re-entry, which is an edge case for most migrations.

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.