duckdb / duckdb/duckdb-spatial

Encoding issue with geospatial data (shapefile)

Open
#394 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
708
Forks
96
Avg merge
1d 21h
Merged PRs (30d)
5

Description

### What happens?

When trying to import a shapefile encoded with CP1252 I have the following error

`InvalidInputException: Invalid Input Error: Invalid unicode (byte sequence mismatch) detected in segment statistics update
`

I tried various options in the st_read but no success

A current workaround is to convert first shapefile to geoparquet with ogr2ogr and then import the geoparquet

### To Reproduce

```
CREATE TABLE t_adresse AS SELECT * FROM st_read('source/t_adresse.shp');
```

Note : the shapefile does have a `.cpg` file providing the encoding

Even forcing encoding do fail:

```
CREATE TABLE t_adresse AS SELECT * FROM st_read('source/t_adresse.shp', open_options=['ENCODING=WINDOWS-1252']);
CREATE TABLE t_adresse AS SELECT * FROM st_read('source/t_adresse.shp', open_options=['ENCODING=CP-1252']);
CREATE TABLE t_adresse AS SELECT * FROM st_read('source/t_adresse.shp', open_options=['ENCODING=CP1252']);

```

However, `SELECT * FROM ST_READ('source/t_adresse.shp')` does not gives error (in python)

Current workaround :

```
ogr2ogr -f parquet t_adresse.parquet source/t_adresse.shp

CREATE TABLE t_adresse AS SELECT * from 't_adresse.parquet'

```

### OS:

MacOS

### DuckDB Version:

1.1

### DuckDB Client:

Python

### Hardware:

_No response_

### Full Name:

Valérian LEBERT

### Affiliation:

Digi-Studio

### What is the latest build you tested with? If possible, we recommend testing with the latest nightly build.

I have tested with a stable release

### Did you include all relevant data sets for reproducing the issue?

No - I cannot easily share my data sets due to their large size

### Did you include all code required to reproduce the issue?

- [X] Yes, I have

### Did you include all relevant configuration (e.g., CPU architecture, Python version, Linux distribution) to reproduce the issue?

- [X] Yes, I have

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.