Clojure v4: on error stop can silently commit a partial load
Nobody has claimed this yet.
- Dominant language
- Common Lisp
- Stars
- 6.5k
- Forks
- 613
- Avg merge
- 3h 46m
- Merged PRs (30d)
- 1
Description
pgloader version
Clojure v4 development build 4.0.0-5ce3726. The affected code is unchanged on current main at ea152ef.
Problem
WITH on error stop can still commit a partial load and exit successfully:
- Database worker tasks are submitted to a Java executor as plain functions, so Clojure dynamic bindings such as
copy/*on-error-stop*, batch size, and prefetch capacity do not reach those workers. - A PostgreSQL COPY
PSQLExceptionalways enters batch bisection/rejection, regardless of strict mode. - MSSQL
ResultSet.nextfailures are treated as end-of-stream, while per-columngetStringfailures are substituted with SQL NULL.
Together, these paths can turn a driver or conversion failure into missing or changed data without a failing process status.
Minimal reproduction
Use a two-row source with values 1 and not-an-integer, and an existing PostgreSQL target whose destination column is integer:
LOAD DATABASE
FROM mssql://user:pass@source:1433/source_db
INTO postgresql://user:pass@target:5432/target_db
WITH data only, create no tables, include no drop,
reset no sequences, workers = 1, on error stop;
Observed result:
- pgloader exits 0.
- The valid row is committed.
- The invalid row is rejected.
- The summary/reject behavior therefore contradicts
on error stop.
An isolated reproduction against PostgreSQL and SQL Server produced exactly that result. With the proposed fix, the same fixture exits 1 and commits zero rows from the failed batch.
Expected behavior
Under on error stop:
- a target COPY error rolls back the active batch and propagates;
- source row-advance and column-read errors propagate;
- the process exits nonzero;
- queued work does not begin another table copy after the failure is observed.
This does not imply a globally transactional migration: batches committed before a later failure remain committed, so failed targets still need to be reset or discarded.
Proposed fix
- Submit executor work using
bound-fnso dynamic settings reach workers. - Rethrow COPY
PSQLExceptionafter rollback when strict mode is active. - Propagate MSSQL JDBC row/column read errors.
- Stop queued table COPY work after the shared failure flag is set.
I have a tested patch and will link the PR here.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue names no files or tests; start by tracing executor submissions and the COPY error path, then inspect MSSQL ResultSet row and column reads. Done means strict mode propagates these failures, rolls back the active batch, exits nonzero, and prevents queued table copies after failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgresql, sql
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100