duckdb / duckdb/duckdb-spatial
Unable to export big files with GDAL to S3
- Dominant language
- C
- Stars
- 708
- Forks
- 96
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 5
Description
### What happens?
Fails CLI (version ) and nodejs with
```
IO Error:
Unknown part number
```
### To Reproduce
```
INSTALL spatial;
LOAD spatial;
INSTALL httpfs;
LOAD httpfs;
DROP SECRET IF EXISTS s3_secret;
CREATE SECRET s3_secret (
TYPE S3,
KEY_ID '***',
SECRET '**',
SESSION_TOKEN '***',
REGION 'us-east-1'
);
COPY (
SELECT * FROM read_parquet([ ... list of urls of big parquet files ...])
)
TO 's3://bucket-name/output/file.geojson'
(FORMAT GDAL, DRIVER 'GeoJSON', LAYER_CREATION_OPTIONS 'WRITE_BBOX=YES', SRS 'EPSG:4326');
```
Changing the FORMAT to:
```
(FORMAT GDAL, DRIVER 'GeoJSON', LAYER_CREATION_OPTIONS 'WRITE_BBOX=NO', SRS 'EPSG:4326');
```
or
```
(FORMAT GDAL, DRIVER 'GeoJSON', SRS 'EPSG:4326');
```
works!
### OS:
aarch64
### DuckDB Version:
v1.4.2 (Andium) 68d7555f68
### DuckDB Client:
CLI and node
### Hardware:
Mac Sillicon Max PRO 32GB
### Full Name:
Antonio Cortés
### Affiliation:
Carto
### Did you include all relevant configuration (e.g., CPU architecture, Linux distribution) to reproduce the issue?
- [x] Yes, I have
### Did you include all code required to reproduce the issue?
- [x] Yes, I have
### Did you include all relevant data sets for reproducing the issue?
No - I cannot easily share my data sets due to their large size
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.