MemberJunction / MemberJunction/MJ
Integration sync: typed query parameter constraints per integration object — filtered sync and partitioned fetch under one abstraction
- Dominant language
- TSQL
- Stars
- 29
- Forks
- 6
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 308
Description
## Problem
An integration object's fetch is all-or-nothing and strictly serial:
1. **All-or-nothing**: there is no way to sync a *subset* of an endpoint. If an object holds millions of rows and only one segment is needed, the sync walks everything. Nothing in metadata expresses "fetch under constraints."
2. **Serial**: one object = one cursor chain. Measured on a multi-million-row Salesforce object (200+ columns) in a production deployment: a fully tuned lane — server-side watermark filtering, per-page durable watermarks, TDS bulk writes, prefetch pipelining — sustains **~20,000 rows/min**, and that is the cursor's natural speed, not the engine's. The write side of the same pipeline benches at ~727k rows/min locally (~2% utilized). The fetch is the entire remaining wall.
3. **Untyped endpoint capabilities**: nothing in metadata records which query parameters an endpoint actually supports, so neither filtering nor partitioning has anything to stand on.
## Proposal (staged)
### v1 — QueryParam constraints (primitive)
A `QueryParam` is a constraint set — `(field, operator, value | range)` — attachable to an entity map / integration object fetch. The engine passes constraints through the existing fetch contract (the same way the watermark value flows today). This immediately enables **filtered sync**: fetch under constraints instead of everything.
### v2 — Typed, verified parameter metadata
Constraints graduate to a **typed attribute of the IntegrationObject** — default empty, later a dedicated table (e.g. `IntegrationObjectQueryParameter`) with per-parameter type, operators, and a **runtime-verified flag**.
Verification matters because vendors silently ignore unknown query parameters — a partition bucket whose filter is ignored returns the *entire* object and destroys the scheme. So a parameter is never used until a one-time runtime probe confirms the endpoint honors it (issue a filtered request, check the result set actually respects the filter, record the verdict in metadata). Endpoint capability discovery becomes mechanical, not documentation archaeology.
### The scheduler: partitioning and filtering are the same abstraction
The engine schedules sub-queries per object:
- a **covering set** of buckets = full sync, fetched in parallel;
- a **chosen subset** = filtered sync.
One abstraction, two features. Concurrency layers become a tree: running integrations → syncing entity maps (the existing lane concurrency) → **sub-queries per endpoint** (new), with every leaf feeding the shared write path.
### The zero-research default dimension
Every filtering connector already implements one range-filterable, ordered parameter: **the watermark field**. Time-slicing over it is an engine-generic partitioner that needs no connector changes and no endpoint research — each bucket is an ordinary fetch with the bucket start as its initial watermark, stopped at the bucket end. Bucket sizing is **adaptive**: per-page watermark stamps give rows-per-time-unit for free, so dense ranges split at runtime (no precomputed distribution needed).
Vendor-native acceleration (e.g. Salesforce Bulk API 2.0 jobs, PK-range scans) slots in later as a connector-specific *implementation* of the same contract — generic floor, specific ceiling.
## Watermark semantics (the subtle part)
Parallel buckets break global ASC ordering, so the durable watermark can no longer be max-seen: each bucket keeps its own frontier, and the object's watermark advances to the lowest fully-covered point (min of completed-bucket ends and in-flight frontiers). The existing periodic durable-floor checkpoint generalizes per bucket.
## Evidence (live, 2026-08)
- Serial-cursor ceiling measured at ~20k rows/min on a multi-million-row object with everything else optimized; the cycle is ~85% fetch wait.
- Fleet behavior confirms cross-object lanes already parallelize cleanly; the only serial unit is the single object's cursor — exactly what sub-queries partition.
## Relationship
Feeds the duplex fetch/write design (#4008): more concurrent fetch streams per object make write-behind and pool-parallel bulk writes proportionally more valuable.
Contributor guide
Research direction
Start by tracing the existing fetch contract, watermark flow, and IntegrationObject/entity-map metadata described in the proposal. Define the QueryParam and scheduler boundaries, then validate filtered and partitioned fetch behavior with durable per-bucket progress; done means supported constraints are verified and a partial fetch cannot silently become a full-object fetch.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- sql
- Domain
- backend, data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100