HarperFast / HarperFast/harper

Add connector plugin contract for external data sync

Open
#509 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

## Context

We would like to support flow into Harper via webhook sync from Salesforce, Zendesk, and Confluence."

Today there is no webhook receiver pattern, no scheduled-sync framework, and no reference connectors. This issue proposes a plugin contract that covers the three modes from the guide: event-driven (webhook), scheduled cadence, and write-through.

## Plugin contract

A connector is a Harper component exporting:

```ts
interface Connector {
name: string; // 'salesforce', 'zendesk', ...
version: string;
configSchema(): JSONSchema;

install(ctx: ConnectorContext): Promise;
uninstall(ctx: ConnectorContext): Promise;

schedule?: CronExpression; // null = no scheduled sync
sync?(ctx, cursor: Cursor): AsyncIterable; // scheduled / backfill
webhook?(ctx, req: Request): AsyncIterable; // event-driven
write?(ctx, change: LocalChange): Promise; // write-through (two-way)
}

type Change = {
resource: string; // target Harper resource
op: 'upsert' | 'delete';
primaryKey: any;
record: Record;
remoteVersion?: string; // for conflict detection
sourceTimestamp: number;
};
```

`ConnectorContext` exposes `scope.resources`, `scope.secrets`, `ctx.checkpoint(cursor)`, `ctx.retry(fn, policy)`, `ctx.dlq(change, reason)`.

## Shared infrastructure provided by Harper

- **`hdb_connector_state`** system table — per-connector cursor, last-success ts, last-error, lag, throughput.
- **`hdb_connector_dlq`** dead-letter table — failed changes with `replay` action.
- **`hdb_connector_conflicts`** table — for `queue-for-review` write-through strategy.
- **Webhook router** — `POST /connectors/:name/webhook` dispatches to `connector.webhook()` with signature verification hook.
- **Scheduler** — extends `core/server/jobs/jobs.js` to invoke `connector.sync()` on cron with cursor passed in.
- **Secret resolution** — connector config references `\${secrets.salesforce.refresh_token}`; resolved at install and on rotation.
- **Retry & rate-limit helpers** — `ctx.retry(fn, { policy: 'expo', jitter: true })`; shared token-bucket rate limiter per connector instance.
- **Observability** — every `Change` emits `analytics.connector_change { name, resource, op, latency_ms, success }`.

## Two-way conflict policy

A write-through connector declares one of:

- `last-write-wins` (default, timestamp comparison)
- `remote-authoritative` (remote wins, local field-merge)
- `queue-for-review` (write to `hdb_connector_conflicts`, no auto-merge)

## CLI

```
harper connectors list
harper connectors install --config
harper connectors backfill --since
harper connectors replay --from-dlq [--filter ...]
harper connectors pause/resume
```

## Reference implementations to ship alongside v1

- `@harperdb/connector-salesforce` — REST + Platform Events CDC
- `@harperdb/connector-zendesk` — REST + Webhooks
- `@harperdb/connector-confluence` — REST + cursor scrape (no webhook API)

These are the three named in the PDF's flagship workload. Each can ship as a separate package/issue.

## Replication interaction

By default, a connector runs as **single-leader with HA failover** on Fabric — exactly one node owns the cursor for a given connector instance. Failover via the existing replication leader election. Two-way writes (`connector.write()`) run on the writing node; cursor updates replicate normally.

## Open decisions

1. Schema mapping — ship recommended schema per connector, allow override via config? Suggested: yes, both.
2. Backfill on install — pull historical data or watch-only? Suggested: flag `--backfill` defaults off; user opts in.
3. Should webhook receivers be a public, generic resource (independent of connectors) for users to roll their own? Suggested: yes — `WebhookResource` base class, connectors are one consumer.

## Acceptance

- [ ] `Connector` interface and `ConnectorContext` defined in core.
- [ ] `hdb_connector_state` / `hdb_connector_dlq` / `hdb_connector_conflicts` system tables.
- [ ] Webhook router with signature-verification hook.
- [ ] Scheduled sync via existing jobs system.
- [ ] CLI commands (list/install/backfill/replay/pause/resume).
- [ ] One reference connector (recommend Zendesk — simplest API surface) lands as a separate package.
- [ ] Documentation: writing a connector, conflict policy choice guide.

## Out of scope (for this issue)

- The reference connectors themselves (separate packages/issues).
- A generic UI for managing connectors (separate from CLI).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.