duckdb / duckdb/duckdb-spatial

Unable to export big files with GDAL to S3

Open
#715 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
708
Forks
96
Avg merge
1d 21h
Merged PRs (30d)
5

Description

### What happens?

Fails CLI (version ) and nodejs with

```
IO Error:
Unknown part number
```

### To Reproduce

```
INSTALL spatial;
LOAD spatial;
INSTALL httpfs;
LOAD httpfs;

DROP SECRET IF EXISTS s3_secret;
CREATE SECRET s3_secret (
TYPE S3,
KEY_ID '***',
SECRET '**',
SESSION_TOKEN '***',
REGION 'us-east-1'
);

COPY (
SELECT * FROM read_parquet([ ... list of urls of big parquet files ...])
)
TO 's3://bucket-name/output/file.geojson'
(FORMAT GDAL, DRIVER 'GeoJSON', LAYER_CREATION_OPTIONS 'WRITE_BBOX=YES', SRS 'EPSG:4326');

```

Changing the FORMAT to:

```
(FORMAT GDAL, DRIVER 'GeoJSON', LAYER_CREATION_OPTIONS 'WRITE_BBOX=NO', SRS 'EPSG:4326');
```
or

```
(FORMAT GDAL, DRIVER 'GeoJSON', SRS 'EPSG:4326');
```
works!

### OS:

aarch64

### DuckDB Version:

v1.4.2 (Andium) 68d7555f68

### DuckDB Client:

CLI and node

### Hardware:

Mac Sillicon Max PRO 32GB

### Full Name:

Antonio Cortés

### Affiliation:

Carto

### Did you include all relevant configuration (e.g., CPU architecture, Linux distribution) to reproduce the issue?

- [x] Yes, I have

### Did you include all code required to reproduce the issue?

- [x] Yes, I have

### Did you include all relevant data sets for reproducing the issue?

No - I cannot easily share my data sets due to their large size

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.