duckdb / duckdb/duckdb-spatial

Overwriting a Shapefile with GDAL COPY leaves tmp sidecars and corrupts the dataset

Open
#859 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
708
Forks
96
Avg merge
1d 21h
Merged PRs (30d)
5

Description

## Description

Overwriting an existing Shapefile using GDAL `COPY` leaves temporary sidecar files behind and can produce a corrupted output dataset.

A Shapefile consists of multiple related files (`.shp`, `.shx`, `.dbf`, etc.). On overwrite, DuckDB writes the new dataset using a `tmp_` prefix but appears to move only the main `.shp` file to its final name. The new sidecars remain under `tmp_*`, while the old sidecars remain paired with the new `.shp`.

I first noticed this while repeatedly running a script with Deno's `--watch` flag, but watch mode is not required to reproduce it.

## Reproduction

Create the output directory:

```sh
mkdir -p /tmp/duckdb-shp-repro
```

Then run:

```sql
INSTALL spatial;
LOAD spatial;

CREATE TABLE shapes (
id INTEGER,
geom GEOMETRY
);

INSERT INTO shapes VALUES
(1, ST_GeomFromText('POINT (0 0)')),
(2, ST_GeomFromText('POINT (1 1)'));

COPY shapes TO '/tmp/duckdb-shp-repro/data.shp'
WITH (
FORMAT GDAL,
DRIVER 'ESRI Shapefile',
LAYER_CREATION_OPTIONS 'ENCODING=UTF-8'
);

DELETE FROM shapes WHERE id = 2;

COPY shapes TO '/tmp/duckdb-shp-repro/data.shp'
WITH (
FORMAT GDAL,
DRIVER 'ESRI Shapefile',
LAYER_CREATION_OPTIONS 'ENCODING=UTF-8'
);
```

After the second export, the directory contains files similar to:

```text
data.cpg
data.dbf
data.shp
data.shx
tmp_data.cpg
tmp_data.dbf
tmp_data.shx
```

Reloading the visible dataset may then fail because `data.shp` belongs to the second export while its sidecars belong to the first:

```sql
SELECT * FROM ST_Read('/tmp/duckdb-shp-repro/data.shp');
```

One observed error was:

```text
IO Error: Error in fread() reading object ... from .shp file
```

## Expected behavior

The second export should replace the complete Shapefile dataset, leave no `tmp_*` files, and `ST_Read` should return one feature.

## Actual behavior

Only the main temporary `.shp` file appears to be moved to its final name. The temporary sidecars remain, leaving the new `.shp` paired with stale sidecars. This can result in both leftover files and an unreadable or internally inconsistent Shapefile.

## Workaround

Explicitly disabling DuckDB's temporary-file mechanism avoids the problem:

```sql
COPY shapes TO '/tmp/duckdb-shp-repro/data.shp'
WITH (
FORMAT GDAL,
DRIVER 'ESRI Shapefile',
LAYER_CREATION_OPTIONS 'ENCODING=UTF-8',
USE_TMP_FILE false
);
```

With `USE_TMP_FILE false`, I confirmed that no `tmp_*` files remain, all Shapefile components are overwritten, and the resulting dataset reloads with the expected single feature.

## Possible cause

DuckDB core's `COPY` finalization appears to treat the output as a single file. `MoveTmpFile` removes the `tmp_` prefix from the requested output path and moves that one file:

https://github.com/duckdb/duckdb/blob/main/src/execution/operator/persistent/physical_copy_to_file.cpp#L3581-L3604

That works for single-file formats, but GDAL drivers such as ESRI Shapefile generate multiple files from one output path.

Potential fixes might include:

- allowing the GDAL copy function to opt out of `USE_TMP_FILE` for multi-file formats;
- teaching the copy lifecycle about all files created by a copy function; or
- having the Spatial extension finalize the complete GDAL dataset.

## Environment

- `@duckdb/node-api`: `1.5.3-r.2`
- OS: macOS, Apple Silicon
- Spatial extension: bundled with the above DuckDB build

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the COPY finalization path in src/execution/operator/persistent/physical_copy_to_file.cpp at the referenced MoveTmpFile code, then reproduce the issue with the SQL steps and GDAL Shapefile output. Trace how the Spatial GDAL copy operation creates and finalizes sidecars. Done means the second export replaces all Shapefile components, leaves no tmp_* files, and ST_Read returns one feature.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, sql
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.