duckdb / duckdb/duckdb-spatial
Encoding issue with geospatial data (shapefile)
- Dominant language
- C
- Stars
- 708
- Forks
- 96
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 5
Description
### What happens?
When trying to import a shapefile encoded with CP1252 I have the following error
`InvalidInputException: Invalid Input Error: Invalid unicode (byte sequence mismatch) detected in segment statistics update
`
I tried various options in the st_read but no success
A current workaround is to convert first shapefile to geoparquet with ogr2ogr and then import the geoparquet
### To Reproduce
```
CREATE TABLE t_adresse AS SELECT * FROM st_read('source/t_adresse.shp');
```
Note : the shapefile does have a `.cpg` file providing the encoding
Even forcing encoding do fail:
```
CREATE TABLE t_adresse AS SELECT * FROM st_read('source/t_adresse.shp', open_options=['ENCODING=WINDOWS-1252']);
CREATE TABLE t_adresse AS SELECT * FROM st_read('source/t_adresse.shp', open_options=['ENCODING=CP-1252']);
CREATE TABLE t_adresse AS SELECT * FROM st_read('source/t_adresse.shp', open_options=['ENCODING=CP1252']);
```
However, `SELECT * FROM ST_READ('source/t_adresse.shp')` does not gives error (in python)
Current workaround :
```
ogr2ogr -f parquet t_adresse.parquet source/t_adresse.shp
CREATE TABLE t_adresse AS SELECT * from 't_adresse.parquet'
```
### OS:
MacOS
### DuckDB Version:
1.1
### DuckDB Client:
Python
### Hardware:
_No response_
### Full Name:
Valérian LEBERT
### Affiliation:
Digi-Studio
### What is the latest build you tested with? If possible, we recommend testing with the latest nightly build.
I have tested with a stable release
### Did you include all relevant data sets for reproducing the issue?
No - I cannot easily share my data sets due to their large size
### Did you include all code required to reproduce the issue?
- [X] Yes, I have
### Did you include all relevant configuration (e.g., CPU architecture, Python version, Linux distribution) to reproduce the issue?
- [X] Yes, I have
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.