dimitri / dimitri/pgloader

Clojure v4: on error stop can silently commit a partial load

Open
#1,774 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Common Lisp
Stars
6.5k
Forks
613
Avg merge
3h 46m
Merged PRs (30d)
1

Description

pgloader version

Clojure v4 development build 4.0.0-5ce3726. The affected code is unchanged on current main at ea152ef.

Problem

WITH on error stop can still commit a partial load and exit successfully:

  1. Database worker tasks are submitted to a Java executor as plain functions, so Clojure dynamic bindings such as copy/*on-error-stop*, batch size, and prefetch capacity do not reach those workers.
  2. A PostgreSQL COPY PSQLException always enters batch bisection/rejection, regardless of strict mode.
  3. MSSQL ResultSet.next failures are treated as end-of-stream, while per-column getString failures are substituted with SQL NULL.

Together, these paths can turn a driver or conversion failure into missing or changed data without a failing process status.

Minimal reproduction

Use a two-row source with values 1 and not-an-integer, and an existing PostgreSQL target whose destination column is integer:

LOAD DATABASE
  FROM mssql://user:pass@source:1433/source_db
  INTO postgresql://user:pass@target:5432/target_db
  WITH data only, create no tables, include no drop,
       reset no sequences, workers = 1, on error stop;

Observed result:

  • pgloader exits 0.
  • The valid row is committed.
  • The invalid row is rejected.
  • The summary/reject behavior therefore contradicts on error stop.

An isolated reproduction against PostgreSQL and SQL Server produced exactly that result. With the proposed fix, the same fixture exits 1 and commits zero rows from the failed batch.

Expected behavior

Under on error stop:

  • a target COPY error rolls back the active batch and propagates;
  • source row-advance and column-read errors propagate;
  • the process exits nonzero;
  • queued work does not begin another table copy after the failure is observed.

This does not imply a globally transactional migration: batches committed before a later failure remain committed, so failed targets still need to be reset or discarded.

Proposed fix
  • Submit executor work using bound-fn so dynamic settings reach workers.
  • Rethrow COPY PSQLException after rollback when strict mode is active.
  • Propagate MSSQL JDBC row/column read errors.
  • Stop queued table COPY work after the shared failure flag is set.

I have a tested patch and will link the PR here.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue names no files or tests; start by tracing executor submissions and the COPY error path, then inspect MSSQL ResultSet row and column reads. Done means strict mode propagates these failures, rolls back the active batch, exits nonzero, and prevents queued table copies after failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, sql
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.