Parquet for MJ — where it fits, where it doesn't, and the plugin-loading change that makes it cheap

## TL;DR

1. **Parquet is already declared in three places in this repo and implemented in zero.** `ArchiveConfiguration.ArchiveFormat` ships `'Parquet'` as a CHECK-constrained enum value that the Archiving engine never reads — setting it silently writes JSON.
2. **Its value to MJ is not query acceleration.** `plans/query-entity-materialization.md` is right that the columnar-warehouse argument addresses scale we don't have. The value is cold-tier retention, interoperability, and reading data that already lives in Parquet.
3. **The change that makes it cheap isn't about Parquet at all.** MJ has both halves of a plugin system — `@RegisterClass`/ClassFactory for *resolution*, the class-registration manifest for *loading* — and they are never joined, so every driver must be a compile-time dependency. The missing link (`DriverImportPath` + a DB-driven `import()`) **already exists in MJ's schema and code, and was abandoned.** Reviving it makes Parquet — and DuckDB, and every future driver — an optional package rather than a core dependency.

Filing for review rather than action. Nothing here is started.

---

## 1. What is already in the repo

| Where | Status |
|---|---|
| `ArchiveConfiguration.ArchiveFormat: 'CSV' \| 'JSON' \| 'Parquet'` — DB column, CHECK constraint, generated ORM, Angular type | **Declared, never read.** `grep -rn "ArchiveFormat" packages/Archiving/` returns **0 hits** |
| `TrainRequest.data_ref` — *"Shared-storage handle to the matrix (Parquet/Arrow)"* (`packages/AI/PredictiveStudio/Core/src/sidecar-contract.ts:143`) | **Typed both ends, refused both ends.** The Python sidecar raises a hard 400: *"Inline `data` is required (data_ref shared-storage is not implemented in v1)"* (`Sidecar/src/python/app/main.py:65-70`) |
| `plans/database-archiving-toolset.md:638` — *"Parquet format: For analytical workloads on archived data"* | Future-phases bullet |

Serialization is hardcoded JSON at three sites (`DefaultArchiveDriver.ts:146`, `:402`, `ArchiveStorageManager.ts:71`), and `.json` is baked into both path builders (`BaseArchiveDriver.ts:66-72`). Setting `ArchiveFormat = 'Parquet'` passes the CHECK, saves cleanly, and is silently ignored. No warning, no error.

**The one primitive a Parquet reader needs, we already built for something else.** `FileStorageBase.GetObjectStream(params)` takes an inclusive `ByteRange { Start, End? }` and returns `ContentRange { Start, End, Total }`, and all seven storage drivers implement it with provider-native ranged reads (S3 `Range` header, Azure `download(offset,count)`, GCS `createReadStream({start,end})`, HTTP Range for Box/Dropbox/Drive/SharePoint). That came out of media streaming. It is exactly the footer-then-column-chunk access pattern Parquet wants.

One asymmetry: `GetObjectParams` (the Buffer-returning `GetObject`) has no `Range` field — only the stream variant does.

---

## 2. Position relative to the MJ data thesis

`plans/query-entity-materialization.md:39` says:

> *"MJ's data thesis holds for our domain: **association data is small.** … The 'you need a columnar warehouse' argument addresses scale we don't have."*

**Nothing in this proposal contradicts that.** Parquet as query acceleration for operational MJ data is solving a problem we don't have; materialized relational snapshots with atomic swap are the right answer at our sizes and are already built.

The three things Parquet is actually for here:

- **Retention economics** on the append-only tables that grow without bound.
- **Interoperability** — a lingua franca for handing data to Python/ML, to customers, to BI.
- **Reach** — reading data that already sits in Parquet without standing up a warehouse.

Parquet as a *file format* is cheap and useful. Parquet as an *architecture* (lakehouse) is the thing the thesis correctly rejects. This proposal is only the former.

---

## 3. Evidence

Measured in a Linux container on Node 22 against synthetic data shaped from our own shipped archive configurations (the `AI Prompt Runs - 6 Month` field list: `Messages`, `Result`, `ErrorMessage`, `ValidationAttempts`, `ArchiveFullRecord: true`). Deterministic seed; synthetic shapes, not production data.

### Size — and the honest result

20,000 archive documents:

```
one pretty JSON file per record (what we write today) 22.70 MB 20,000 objects
NDJSON, one file, gzip -6 1.13 MB 1 object
Parquet, one file, snappy + statistics 1.52 MB 1 object (15.0x smaller than today)
```

**Gzipped NDJSON beats Parquet on raw size.** If all we want is a smaller archive, batch-and-gzip is simpler and wins. Size is not the argument.

### Selective read — this is the argument

```
Which archived runs errored? Parquet 4,148 B (0.3% of file) vs gzip NDJSON 1,183,272 B -> 285x less I/O
Error rate by agent, with error text Parquet 12,177 B (0.8% of file) vs -> 97x less I/O
Token + cost totals by model Parquet 53,480 B (3.4% of file) vs -> 22x less I/O
Rehydrate the full archived payload Parquet 100% of file -> 1x (no win)
```

Gzipped NDJSON must be fully decompressed to answer any of them. This turns cold archive from write-only storage into queryable storage — which is the unimplemented "Archive search" item at `plans/database-archiving-toolset.md:637`.

### At realistic volume

2M synthetic Record Change rows:

```
COPY TO hive-partitioned by year/month, zstd 6,521 ms -> 80.7 MB across 24 partition files

scan ALL partitions, count by type 23 ms
ONE month via hive partition predicate 3.7 ms
ONE month via row-group stats (no partition col) 8.9 ms
COUNT(DISTINCT RecordID) over 2M rows 103 ms
```

### Range reads through a storage-shaped API

I wrote an adapter over a mock with only `GetObjectMetadata` + `GetObjectStream({Range})` — about 10 lines, because hyparquet's `AsyncBuffer` is `{ byteLength, slice(start,end) }`, which is what `FileStorageBase` already exposes. Against a 78.7 MB / 2M-row file:

```
footer read 1 GET 512 KB 0.64% of object
count errors (1 column) 10 GETs ~0 MB 0.0% of object
changes by type (2 columns) 20 GETs 0.07 MB 0.1% of object
audit trail including RecordID 10 GETs 39.29 MB 49.9% of object
```

Two lessons. **UUID columns destroy the economics** — `RecordID` as 36-char VARCHAR is ~50% of the file, high-entropy and barely compressible, so any query touching it reads half the object. Store UUIDs as `FIXED_LEN_BYTE_ARRAY(16)` with the UUID logical type. And **on small files ranged reads lose outright** — on a 1.5 MB file the speculative tail read plus column fetches came to **111% of the object**, worse than downloading it. Range reads pay off above roughly 50 MB.

### Why this must never touch a write path

5,000 Record Changes, one Parquet file per row vs one batched file:

```
(A) one file per row 5000 files 5.22 MB written in 1555 ms
(B) one batched file 1 file 0.05 MB written in 55 ms

-> 103x the bytes, 28x the write time, 5000 object-storage PUTs instead of 1
query COUNT(*): 299 ms over 5000 files vs 2 ms over 1 file -> 130x slower
```

On a local filesystem. On S3 each of those 5,000 files is a separate GET.

### Parquet is not mutable

```
DELETE on a parquet-backed view -> Binder Error: Can only delete from base table
INSERT on a parquet-backed view -> Catalog Error: v is not an table
```

The only write is `COPY (SELECT …) TO`. No append, no update, no delete. Everything that makes Parquet look mutable (Iceberg, Delta) is a separate table format maintaining a log of immutable files.

---

## 4. The architectural change — join the two halves of the plugin system

This is the part worth reviewing even if we never ship Parquet.

### MJ has both halves and never joins them

1. **`@RegisterClass` + `MJGlobal.ClassFactory`** — the *resolution* half. Universal and genuinely metadata-driven: a DB column names a key, the factory resolves it. EDS (`ExternalDataSourceRouter.ts:93-112`), storage (`MJStorage/src/util.ts:62-88`), archiving, AI vendors, auth providers, scheduled jobs, actions, integration connectors all work this way.
2. **The class-registration manifest** (`mj codegen manifest` → `ServerBootstrap/src/generated/mj-class-registrations.ts`, 2,691 lines, 110 packages) — the *loading* half. It is **100% compile-time**: `GenerateClassRegistrationsManifest.ts:268-328` seeds `walkDependencyTree` from `package.json` `dependencies`/`devDependencies` and BFS-walks transitive deps, emitting static named imports. **No config input, no database input.** A package outside the dependency closure is invisible.

**The gap: `DriverClass` selects among already-loaded classes; it never causes a load.** Every driver package must be a compile-time dependency of `ServerBootstrap` (or the host app).

### The missing link already exists — and was abandoned

`DriverImportPath` is a real column on **five** entities: `MJ: AI Models`, `MJ: AI Model Vendors`, `MJ: Company Integrations`, `MJ: Encryption Key Sources`, `MJ: Queue Types`. And `AIEngine.getDriver()` (`packages/AI/Engine/src/AIEngine.ts:1556-1570`) implements the complete pattern:

```ts
const driverClassName = model.DriverClass;
const driverModuleName = model.DriverImportPath;
if (driverModuleName && driverModuleName.length > 0) {
const driverModule = await import(driverModuleName);
if (!driverModule) throw new Error(`Error loading driver module '${driverModuleName}'`);
}
return MJGlobal.Instance.ClassFactory.CreateInstance(BaseModel, driverClassName, apiKey);
```

**This is the only place in the repo where a database column decides which npm package to import.** It sits on a `@deprecated` AI Actions path; the live path (`AIPromptRunner` / `AIModelRunner`) never calls it and relies on the manifest instead. `Encryption Key Sources` has `DriverImportPath` populated in metadata (`"@memberjunction/encryption"`) and **never read** — `EncryptionEngine.ts:865-900` uses `DriverClass` only.

So the convention was provisioned, then abandoned in favour of the manifest. And the three catalogs that most need it never got the column at all: `MJ: External Data Source Types`, `MJ: File Storage Providers`, `MJ: Archive Configuration Entities`.

*(Correction to the first version of this issue: I originally wrote that no metadata table carries a package name. That was wrong — I searched for `DriverPackage`/`PackageName`/`NpmPackage`; the actual column is `DriverImportPath`, 35 occurrences across those five entities. The finding makes the proposal stronger: this is reviving a precedented pattern, not inventing one.)*

### What the gap costs today

1. **Hardcoded import lists — four places per driver family.** Adding an 8th EDS driver means editing: `packages/CodeGenLib/src/Database/manage-metadata.ts:2755-2763` (8 `await import(...)` calls), `CodeGenLib/package.json:54-60` (7 deps), `ServerBootstrap/src/generated/mj-class-registrations.ts` (named imports at `:927-965`, the anti-tree-shake array at `:2180-2186`, the package list at `:2644-2651`), and `ServerBootstrap/package.json:118-124` (7 deps). `guides/EXTERNAL_DATA_SOURCES_GUIDE.md`'s "adding a driver" recipe mentions none of this, so a new driver appears to work at runtime and silently fails to introspect under CodeGen. Every CodeGen run touching external entities also pays to load pg + mongodb + oracle + databricks even when one source is configured.
2. **Bundled dependencies.** `@memberjunction/storage` carries every vendor SDK as a hard dependency — `@aws-sdk/client-s3`, `@azure/storage-blob`, `@google-cloud/storage`, `googleapis`, `box-node-sdk`, `dropbox`, 4 × `@microsoft/*`. Installing MJ pulls all seven clouds whether or not you use one. The only escape hatch is `ServerBootstrapLite`'s build-time `--exclude-packages @memberjunction/storage`.
3. **A hardcoded `switch` with no ClassFactory at all.** `MJExportEngine` has `ExportFormat = 'excel' | 'csv' | 'json'` (`types.ts:4`) dispatched by duplicated switches (`export-engine.ts:115-128`, `:134-146`), no `@RegisterClass`, and no `@memberjunction/global` dependency. The union leaks into six packages and the Explorer UI, so a new format means editing every consumer. **This subsystem needs the full treatment — entity, base class, and registration; it has none of the three.**

### The best existing precedent is Open Apps + `@memberjunction/dynamic-packages`

`@memberjunction/dynamic-packages` is a *"process-agnostic loader for packages whose names are only known at runtime"* whose stated purpose is firing `@RegisterClass` side effects. It has per-process scoping (`Processes`/`ExcludeProcesses`, `CliProcessId`), per-tier filtering, a policy switch, an `MJ_DYNAMIC_PACKAGES=none` escape, and a robustness contract: per-package try/catch, unresolvable packages reported as not-found, **boot never crashes because of an app package**.

The `mj` CLI **already calls it for every non-light command, including `codegen`** — `packages/MJCLI/src/hooks/prerun.ts:146-147` → `loadDynamicPackagesForCommand(commandId)`. Its docstring records the ordering:

> *"Heavy commands then load the installed Open Apps' server packages … **AFTER the manifest**, so an app's `@RegisterClass` wins via load-order priority, and **BEFORE the command opens a database provider**, the same ordering MJAPI uses."*

The process ID is `cli:`, so `cli:codegen` is already addressable as a scoping target. **CodeGen's eight `await import()` calls are redundant with a loader that already ran in the same process, before the provider opened.**

Its flagship consumer is the **Integrations repo**: 36 connectors moved out of this monorepo into `MemberJunction/Integrations`, each its own Open App with its own package, migrations and CI. `packages/Integration/connectors/` retains only three abstract base classes. Each app's seed migration rewrites the pre-existing `__mj.Integration` row in place (same hardcoded ID), setting `ClassName` and `ImportPath`. Worth noting for our design: `ConnectorFactory.ts:23-59` reads `ClassName` and **never reads `ImportPath`** — the package is loaded by dynamic-packages from `mj.config.cjs`, and the two are kept consistent only by a four-way identity invariant (`@RegisterClass` key ≡ `ClassName` ≡ `ImportPath` ≡ npm package name).

Two secondary precedents worth stealing from:

- **Server Extensions** (`ServerExtensionsCore/src/ServerExtensionLoader.ts:240-310`): entries come from `mj.config.cjs` **or** from a package's own `package.json` `memberjunction.serverExtensions` block. This is the closest thing MJ has to *self-describing* plugin metadata — the package declares its own capability rather than the host enumerating it.
- **`AuthProviderFactory.ts:7-20`**, which records that a hardcoded provider roster was deliberately deleted, concluding: *"Adding a provider is therefore: ship a `@RegisterClass(BaseAuthProvider, 'x')` subclass (in this package, **or in any package covered by a class-registration manifest**) and add a row naming 'x' as its DriverClass. No edit here."* That parenthetical is both the thesis of the current architecture and its limitation — "covered by a manifest" means "a compile-time dependency."

### Proposed shape

- **Keep the manifest as the fast path** for in-tree drivers; add runtime load as the fallback. This is already how `resolveStorageDriver`'s error message frames the two failure modes (`MJStorage/src/util.ts:84-86`).
- **CodeGen drops its hardcoded list** and resolves the router from ClassFactory as it already does, relying on the prerun load that has already happened.
- **`MJExportEngine` converts to the pattern MJStorage and EDS already use** — `@RegisterClass(BaseExporter, format)` replacing both switches, `getSupportedFormats()` reading registrations, and an entity to back it.
- **Where package names live is the open design question** (see §10). Two candidates: (a) config-only via `dynamicPackages`, matching Open Apps and Integrations exactly; (b) revive `DriverImportPath`, add it to the three catalogs missing it, and give it DynamicPackages' robustness contract rather than `AIEngine`'s bare `await import()`.
- **The security argument leans toward config.** A package name in a DB row is a code-execution vector — anyone who can write metadata could name any npm package. Today package names come from files controlled by whoever deploys. There is no allowlist concept in `DynamicPackages` or `Config`, so option (b) needs one and option (a) does not.

### What that buys Parquet

Parquet becomes an optional package rather than a core dependency, which turns the engine question into a deployment choice instead of an architectural one — relevant because `@duckdb/node-api` is ~70 MB with a native binary. A deployment that wants SQL over a lake installs the DuckDB-backed package; one that only needs archive read-back installs the zero-dependency pure-JS one; one that needs neither installs nothing.

**Related prior direction:** `plans/p6-viewtype-plugin-design-from-amith.md` and `plans/view-type-full-plugin-migration.md` describe the same migration at the UI tier — view types as dynamically-mounted `IViewRenderer` plugins replacing hardcoded blocks and a `driverClassToViewMode()` bridge. Same disease, same cure, different tier; worth aligning vocabulary.

---

## 5. Where Parquet pays, ranked

**① Archive format.** Eight archive configurations ship in `metadata/archiving/` — Record Changes (12mo), AI Prompt Runs (6mo), AI Agent Runs, AI Agent Run Steps, Audit Logs, Action Execution Logs, Scheduled Job Runs, Communication Logs. Meanwhile only one scheduled retention job exists (`ActionLogRetentionScheduledJobDriver`); there is no retention for prompt runs, agent runs, audit logs, or record changes.

The real work is not a format branch. The current layout is `{Root}/{Entity}/{RecordID}/{VersionStamp}.json` — **one file per record per version**, the worst possible granularity for Parquet (see the 103x above). Honoring `ArchiveFormat` means changing the **unit of archival** from record→object to batch→object, with `ArchiveRunDetail` pointing at `(file, row-group)`.

That refactor fixes an existing bug incidentally: cascade batch files written under `HardDelete` are never logged to `ArchiveRunDetail` with their own `StoragePath`, so hard-deleted children are currently **archived and unrecoverable**.

**② `TrainRequest.data_ref`.** The seam is designed, typed, documented and deliberately deferred on both ends. The current path `JSON.stringify`s the whole matrix with a `Content-Length` header, no streaming, and ships a second full holdout matrix on the same request. Embeddings arrive as individual `emb_0…emb_767` columns — 768 JSON numbers per row. pandas/polars/pyarrow read Parquet natively, so this is implementation with no design work.

**③ `DataContextItem.DataJSON`.** `packages/MJDataContext/src/types.ts:707` does `JSON.stringify(item.Data)` into an `nvarchar(MAX)` column — an entire result set as a JSON string in a database row. Every load path sets `IgnoreMaxRows: true` (`types.ts:318, 349, 601`), explicitly defeating `UserViewMaxRows`, and `full_entity` means all rows. A storage handle + Parquet body is a correctness improvement as much as a size one.

**④ A lake-reading EDS driver.** Cheaper than it looks: `FilterDialect: 'ansi'` already passes `CK_ExternalDataSourceType_FilterDialect` (values are `tsql, ansi, pgsql, mysql, oracle, mongo-ast`), `PagingStrategy: 'LimitOffset'` fits, and no migration is needed. The MySQL driver is 320 LOC as a size signal. All the SQL shapes a driver generates work well over Parquet — `SELECT/WHERE/ORDER BY/LIMIT OFFSET` 11 ms, `COUNT(*)` 1.9 ms (reads footer stats, no scan), `GROUP BY` 4.8 ms, `DESCRIBE` gives introspection.

**Check first whether we need it.** `Databricks SQL Warehouse (External)` and `Microsoft Fabric SQL Endpoint (External)` both ship today. Delta Lake is Parquet plus a transaction log; a Fabric Lakehouse SQL endpoint is Parquet in OneLake. If a customer's Parquet is already in either, **MJ reads it today with zero code**.

**⑤ Export format.** `ExportResult.data` is already `Uint8Array`, so the return shape fits unchanged. Blocked on §4 plus a real physical type model — `ColumnDataType` is a formatting hint (`'currency' | 'percentage'`), not a type, and `export()` buffers the whole dataset.

---

## 6. Where it does not fit

- **Operational query acceleration.** Answered by materialization (#2770). Not reopening it.
- **`RecordSetProcessor`.** Already streams correctly (keyset paging, `NextBatch(cursor, batchSize)`). Leave it.
- **Searchable vector embeddings.** `EntityRecordDocument.VectorJSON` storing embeddings as JSON text is wasteful, but `plans/colocated-vector-search.md` (pgvector / SQL Server 2025 `VECTOR(N)`) is the better direction and is already in flight.
- **Writing through External Data Sources.** EDS is read-only by architecture, across four enforcement layers. `ReadOnlyExternalBaseEntity.ts:9-12` states: *"MJ never owns their write path because transactions, **Record Changes**, and row-level security cannot be guaranteed across heterogeneous remote systems. Write support is an explicit non-goal."* `ExternalDataSourceType.SupportsReadWrite` exists, defaults 0, and its extended property reads *"Reserved for a future write-capable phase."* Nothing reads it. This proposal does not ask to change that.
- **Record Changes specifically.** Not because the write is transactional — on SQL Server it isn't (see §7) — but because **other things point at those rows**: `VersionLabelItem.RecordChangeID` is a real FK `INNER JOIN`ed in `vwVersionLabelItems`; `RecordChange.RestoredFromID` is a self-referencing FK walked by a recursive CTE (`fnRecordChangeRestoredFromID_GetRootID`) for restore lineage; and `ExternalChangeDetection` does latest-change-per-`(EntityID, RecordID)` point lookups, the worst access pattern for a columnar file. You cannot satisfy a foreign key from a file.

**"And the like" is the better target than Record Changes.** Three of the four other high-volume writers are already fire-and-forget through `BaseEntitySaveQueue` (`packages/MJCore/src/generic/BaseEntitySaveQueue.ts:22`), whose header says saves *"are fire-and-forget and never throw outward"*:

| Writer | Today | Coupling |
|---|---|---|
| AIPromptRun | `_promptRunQueue = new BaseEntitySaveQueue(...)` | none |
| AIAgentRun / Steps | `_stepSaveQueue = new AgentRunStepSaveQueue()` | none |
| ActionExecutionLog | `_logQueue.Insert(...)`, already has a `None`/`FailuresOnly` gate and per-row `RetentionPeriod` | none |
| AuditLog | awaited, but wrapped in try/catch that logs and returns `null` | none |
| RecordChange | inline in the save batch, `@ID`-dependent | welded |

All carry `TrackRecordChanges: false`. These are the volume, and they are trivially divertible.

---

## 7. Three unrelated defects found along the way

Worth tickets regardless of anything above.

**Record Changes are not atomic on SQL Server.** There is no `BEGIN TRANSACTION` and no `SET XACT_ABORT ON` in the batch `WrapSaveCallWithRecordChange` emits; a transaction attaches only if one is already ambient, which for a flat entity saved over GraphQL never happens. So the save and the audit row are two autocommit transactions sharing one round trip. If the record-change EXEC fails, the row is **already committed**, `ExecuteSQL` throws, and the caller is told the save failed — row persisted, audit row missing, caller misinformed, silently. PostgreSQL is genuinely atomic; it fuses both into one CTE statement (`PostgreSQLDataProvider.ts:872-882`).

**Per-save overhead in the record-change path.** `FullRecordJSON` is a complete serialized copy of the row on every tracked save, and on SQL Server the MAX blobs are inlined as SQL string literals rather than parameters — so every tracked save ships a serialized row inside the SQL text, defeating plan reuse. There is also a per-write `Entity`-name lookup and a discarded `SELECT * FROM vwRecordChanges` read-back through a 5-join view containing a recursive CTE.

**`ArchiveProcessor.ResolveDriver` has a fallback that can never fire.** `packages/Archiving/Engine/src/ArchiveProcessor.ts:83-95` calls `ClassFactory.CreateInstance` and guards with `if (driver)`, intending to fall back to `DefaultArchiveDriver` on a miss. But `CreateInstance` returns an instance of the *abstract base* rather than null when unregistered, so the guard is effectively always true — a typo'd or unloaded `DriverClass` yields a hollow base object instead of the fallback. The EDS router (`ExternalDataSourceRouter.ts:101-112`) and `resolveStorageDriver` both added explicit `GetRegistration` pre-checks for exactly this; Archiving did not. This gets worse, not better, once drivers load dynamically.

---

## 8. Constraints and traps

- **Right-to-erasure.** A Parquet archive of PII plus a deletion request means rewriting files; there is no DELETE. Partition so a deletion is a bounded rewrite — a compliance decision, not a performance one.
- **We will want a catalog and should not adopt Iceberg to get one.** `ArchiveConfiguration` / `ArchiveRun` / `ArchiveRunDetail` already are one.
- **Small files kill it.** Target ≥64 MB files / ≥100k-row row groups.
- **Codec disagreement is silent.** `hyparquet` cannot read ZSTD without `hyparquet-compressors`; DuckDB defaults to ZSTD; the pure-JS writer emits SNAPPY only. Pick one and write it down.
- **DuckDB downloads extensions at runtime.** `httpfs` and `postgres_scanner` both failed to install in a network-restricted container. Statically available: `parquet`, `json`, `icu`, `core_functions`, `autocomplete`. Local Parquet works out of the box; reading S3 directly does not.
- **`@duckdb/node-api` does not expose `registerFileBuffer`**, so DuckDB cannot be handed bytes we already fetched from MJStorage — it needs a temp-file spill of the whole object.
- **The EDS driver contract assumes a named remote object with introspectable columns** (`ResolveObjectName`, `IntrospectSchema`). A bare S3 prefix of Parquet files is not a first-class shape, and that mapping has no home in metadata today. This is the genuinely unsolved part of ④.
- **EDS limits worth knowing before committing**: `HARD_MAX_EXTERNAL_ROWS = 50_000` on the RunView path applied even to an explicit `MaxRows` (default 1,000; per-source override via `ConnectionConfig {"maxRowLimit": N}`); the RunQuery path escapes both that ceiling and RLS; plain server-side `RunView` does not cache external results; Angular forms are generated with no external special-casing, so users get an editable-looking form whose Save returns `false`; Datasets hard-fail; relationship loading is silently skipped on `Load`; `FullTextSearchEnabled` returns nothing with no error surfaced.
- **`DataSource: 'Materialized'` is unreachable for external entities** — the external dispatch returns before the materialized-view swap, and external+RLS materialization is refused at mint and DriftHold'd.
- **There is no union/tiering concept.** "Recent rows from SQL + older from Parquet as one view" does not exist. Two entities plus a client-side merge is the cheap answer and matches MJ's stated preference for flat `RunViews` + in-memory joins.

### Library landscape (verified versions)

| Package | Version | Notes |
|---|---|---|
| `hyparquet` | 1.30.1 | MIT, **zero dependencies**, Node + browser, `AsyncBuffer` matches `FileStorageBase` ranges |
| `hyparquet-writer` | 0.16.9 | MIT, depends only on hyparquet; SNAPPY only |
| `hyparquet-compressors` | 1.1.1 | MIT; needed for ZSTD |
| `@duckdb/node-api` | 1.5.5-r.5 | MIT, ~70 MB, native binary; glibc — fine for our Debian-based images |
| `parquet-wasm` | 0.7.2 | Rust arrow-rs via WASM; full read+write |
| `apache-arrow` | 21.2.0 | v13 already in the lockfile transitively via `@databricks/sql` |

---

## 9. Suggested phasing

1. **Plugin loading (§4)** — independent of Parquet, and the thing that makes everything else additive. Drop CodeGen's hardcoded list; convert `MJExportEngine` to the ClassFactory pattern; fix the `ResolveDriver` guard first, since dynamic loading makes that failure mode more likely.
2. **Check Databricks/Fabric.** If that is where customer Parquet lives, ④ is configuration, not development.
3. **Archive batching + `ArchiveFormat`** — starting with the AI/Action logs, not Record Changes. Fixes the unrecoverable-cascade bug on the way.
4. **`data_ref`** — smallest well-defined unit of work in the list.
5. **A lake-reading EDS driver** — only if we need raw-lake reads rather than a warehouse over one, and only after the object-naming problem has an answer.

## 10. Open questions

- **Config-declared or metadata-declared package names?** Open Apps and Integrations do the former; `AIEngine.getDriver` does the latter. If metadata, we need an allowlist that does not exist today.
- Should we revive `DriverImportPath` and extend it to EDS Types / File Storage Providers / Archive Configuration Entities, or retire it in favour of `dynamicPackages`? Leaving it half-wired on five entities is the worst of both.
- Should `ArchiveFormat` move from an enum to a registration key once formats are pluggable?
- Do we want archive read-back as a first-class queryable entity, or is `ArchiveRecovery.RestoreVersion` sufficient?
- Is the SQL Server record-change atomicity gap a bug to fix or documented behaviour?

## Reproducing the measurements

All figures come from standalone scripts against `hyparquet` / `hyparquet-writer` / `@duckdb/node-api`, using synthetic data generated from the field lists in `metadata/archiving/.archive-configurations.json` with a deterministic seed. Happy to attach the scripts if useful for review.

## References

- #2449 — External Data Sources (merged) · #2770 — Query & Entity Materialization design plan (merged)
- `plans/query-entity-materialization.md` · `plans/database-archiving-toolset.md` · `plans/predictive-studio.md` · `plans/p6-viewtype-plugin-design-from-amith.md` · `plans/view-type-full-plugin-migration.md`
- `guides/EXTERNAL_DATA_SOURCES_GUIDE.md` · `guides/DYNAMIC_PACKAGE_LOADING_GUIDE.md` · `guides/SERVER_EXTENSIONS_GUIDE.md`
- `packages/Integration/connectors/README.md` (the Integrations-repo split)

Start by reading AIEngine.ts:1556-1570, EncryptionEngine.ts:865-900, and GenerateClassRegistrationsManifest.ts:268-328 to compare dynamic imports with manifest loading. Then inspect the three catalogs identified as missing DriverImportPath and the listed package and guide changes. The issue is filed for review and does not define a single implementation scope or acceptance criteria yet.

Written by the indexing model from the issue text.