MemberJunction / MemberJunction/MJ

Integration sync: typed query parameter constraints per integration object — filtered sync and partitioned fetch under one abstraction

Open
#4,007 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TSQL
Stars
29
Forks
6
Avg merge
1d 8h
Merged PRs (30d)
308

Description

## Problem

An integration object's fetch is all-or-nothing and strictly serial:

1. **All-or-nothing**: there is no way to sync a *subset* of an endpoint. If an object holds millions of rows and only one segment is needed, the sync walks everything. Nothing in metadata expresses "fetch under constraints."
2. **Serial**: one object = one cursor chain. Measured on a multi-million-row Salesforce object (200+ columns) in a production deployment: a fully tuned lane — server-side watermark filtering, per-page durable watermarks, TDS bulk writes, prefetch pipelining — sustains **~20,000 rows/min**, and that is the cursor's natural speed, not the engine's. The write side of the same pipeline benches at ~727k rows/min locally (~2% utilized). The fetch is the entire remaining wall.
3. **Untyped endpoint capabilities**: nothing in metadata records which query parameters an endpoint actually supports, so neither filtering nor partitioning has anything to stand on.

## Proposal (staged)

### v1 — QueryParam constraints (primitive)

A `QueryParam` is a constraint set — `(field, operator, value | range)` — attachable to an entity map / integration object fetch. The engine passes constraints through the existing fetch contract (the same way the watermark value flows today). This immediately enables **filtered sync**: fetch under constraints instead of everything.

### v2 — Typed, verified parameter metadata

Constraints graduate to a **typed attribute of the IntegrationObject** — default empty, later a dedicated table (e.g. `IntegrationObjectQueryParameter`) with per-parameter type, operators, and a **runtime-verified flag**.

Verification matters because vendors silently ignore unknown query parameters — a partition bucket whose filter is ignored returns the *entire* object and destroys the scheme. So a parameter is never used until a one-time runtime probe confirms the endpoint honors it (issue a filtered request, check the result set actually respects the filter, record the verdict in metadata). Endpoint capability discovery becomes mechanical, not documentation archaeology.

### The scheduler: partitioning and filtering are the same abstraction

The engine schedules sub-queries per object:

- a **covering set** of buckets = full sync, fetched in parallel;
- a **chosen subset** = filtered sync.

One abstraction, two features. Concurrency layers become a tree: running integrations → syncing entity maps (the existing lane concurrency) → **sub-queries per endpoint** (new), with every leaf feeding the shared write path.

### The zero-research default dimension

Every filtering connector already implements one range-filterable, ordered parameter: **the watermark field**. Time-slicing over it is an engine-generic partitioner that needs no connector changes and no endpoint research — each bucket is an ordinary fetch with the bucket start as its initial watermark, stopped at the bucket end. Bucket sizing is **adaptive**: per-page watermark stamps give rows-per-time-unit for free, so dense ranges split at runtime (no precomputed distribution needed).

Vendor-native acceleration (e.g. Salesforce Bulk API 2.0 jobs, PK-range scans) slots in later as a connector-specific *implementation* of the same contract — generic floor, specific ceiling.

## Watermark semantics (the subtle part)

Parallel buckets break global ASC ordering, so the durable watermark can no longer be max-seen: each bucket keeps its own frontier, and the object's watermark advances to the lowest fully-covered point (min of completed-bucket ends and in-flight frontiers). The existing periodic durable-floor checkpoint generalizes per bucket.

## Evidence (live, 2026-08)

- Serial-cursor ceiling measured at ~20k rows/min on a multi-million-row object with everything else optimized; the cycle is ~85% fetch wait.
- Fleet behavior confirms cross-object lanes already parallelize cleanly; the only serial unit is the single object's cursor — exactly what sub-queries partition.

## Relationship

Feeds the duplex fetch/write design (#4008): more concurrent fetch streams per object make write-behind and pool-parallel bulk writes proportionally more valuable.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the existing fetch contract, watermark flow, and IntegrationObject/entity-map metadata described in the proposal. Define the QueryParam and scheduler boundaries, then validate filtered and partitioned fetch behavior with durable per-bucket progress; done means supported constraints are verified and a partial fetch cannot silently become a full-object fetch.

Written by the indexing model from the issue text.

Assessment

Tech stack
sql
Domain
backend, data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.