HarperFast / HarperFast/harper
Add connector plugin contract for external data sync
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 200
Description
## Context
We would like to support flow into Harper via webhook sync from Salesforce, Zendesk, and Confluence."
Today there is no webhook receiver pattern, no scheduled-sync framework, and no reference connectors. This issue proposes a plugin contract that covers the three modes from the guide: event-driven (webhook), scheduled cadence, and write-through.
## Plugin contract
A connector is a Harper component exporting:
```ts
interface Connector {
name: string; // 'salesforce', 'zendesk', ...
version: string;
configSchema(): JSONSchema;
install(ctx: ConnectorContext): Promise;
uninstall(ctx: ConnectorContext): Promise;
schedule?: CronExpression; // null = no scheduled sync
sync?(ctx, cursor: Cursor): AsyncIterable; // scheduled / backfill
webhook?(ctx, req: Request): AsyncIterable; // event-driven
write?(ctx, change: LocalChange): Promise; // write-through (two-way)
}
type Change = {
resource: string; // target Harper resource
op: 'upsert' | 'delete';
primaryKey: any;
record: Record;
remoteVersion?: string; // for conflict detection
sourceTimestamp: number;
};
```
`ConnectorContext` exposes `scope.resources`, `scope.secrets`, `ctx.checkpoint(cursor)`, `ctx.retry(fn, policy)`, `ctx.dlq(change, reason)`.
## Shared infrastructure provided by Harper
- **`hdb_connector_state`** system table — per-connector cursor, last-success ts, last-error, lag, throughput.
- **`hdb_connector_dlq`** dead-letter table — failed changes with `replay` action.
- **`hdb_connector_conflicts`** table — for `queue-for-review` write-through strategy.
- **Webhook router** — `POST /connectors/:name/webhook` dispatches to `connector.webhook()` with signature verification hook.
- **Scheduler** — extends `core/server/jobs/jobs.js` to invoke `connector.sync()` on cron with cursor passed in.
- **Secret resolution** — connector config references `\${secrets.salesforce.refresh_token}`; resolved at install and on rotation.
- **Retry & rate-limit helpers** — `ctx.retry(fn, { policy: 'expo', jitter: true })`; shared token-bucket rate limiter per connector instance.
- **Observability** — every `Change` emits `analytics.connector_change { name, resource, op, latency_ms, success }`.
## Two-way conflict policy
A write-through connector declares one of:
- `last-write-wins` (default, timestamp comparison)
- `remote-authoritative` (remote wins, local field-merge)
- `queue-for-review` (write to `hdb_connector_conflicts`, no auto-merge)
## CLI
```
harper connectors list
harper connectors install --config
harper connectors backfill --since
harper connectors replay --from-dlq [--filter ...]
harper connectors pause/resume
```
## Reference implementations to ship alongside v1
- `@harperdb/connector-salesforce` — REST + Platform Events CDC
- `@harperdb/connector-zendesk` — REST + Webhooks
- `@harperdb/connector-confluence` — REST + cursor scrape (no webhook API)
These are the three named in the PDF's flagship workload. Each can ship as a separate package/issue.
## Replication interaction
By default, a connector runs as **single-leader with HA failover** on Fabric — exactly one node owns the cursor for a given connector instance. Failover via the existing replication leader election. Two-way writes (`connector.write()`) run on the writing node; cursor updates replicate normally.
## Open decisions
1. Schema mapping — ship recommended schema per connector, allow override via config? Suggested: yes, both.
2. Backfill on install — pull historical data or watch-only? Suggested: flag `--backfill` defaults off; user opts in.
3. Should webhook receivers be a public, generic resource (independent of connectors) for users to roll their own? Suggested: yes — `WebhookResource` base class, connectors are one consumer.
## Acceptance
- [ ] `Connector` interface and `ConnectorContext` defined in core.
- [ ] `hdb_connector_state` / `hdb_connector_dlq` / `hdb_connector_conflicts` system tables.
- [ ] Webhook router with signature-verification hook.
- [ ] Scheduled sync via existing jobs system.
- [ ] CLI commands (list/install/backfill/replay/pause/resume).
- [ ] One reference connector (recommend Zendesk — simplest API surface) lands as a separate package.
- [ ] Documentation: writing a connector, conflict policy choice guide.
## Out of scope (for this issue)
- The reference connectors themselves (separate packages/issues).
- A generic UI for managing connectors (separate from CLI).
Contributor guide
Assessment
This issue has not been assessed yet.