duckdb / duckdb/duckdb-spatial

ST_Read fails on non-UTF-8 Shapefile (macOS only?)

Open
#744 4 comments 1 reaction 0 assignees View on GitHub
GDAL
Dominant language
C
Stars
708
Forks
96
Avg merge
1d 21h
Merged PRs (30d)
5

Description

I found `ENCODING=` GDAL option doesn't work. I don't remember if this used to work on macOS, but at least this works on Windows. So, maybe something is wrong with macOS build?

```sql
FROM ST_Read('/path/to/data.shp', open_options = ["ENCODING=CP932"])
```

The above doesn't fail and silently returns the results like below.

```
┌────────────┬────────────┬────────────┐
│ A22_000001 │ A22_000002 │ A22_000003 │
│ varchar │ varchar │ varchar │
├────────────┼────────────┼────────────┤
│ ????INVA… │ ????INVA… │ ????INVA… │
│ ????INVA… │ ????INVA… │ ????INVA… │
│ ????INVA… │ ????INVA… │ ????INVA… │
│ ????INVA… │ ????INVA… │ ????INVA… │
│ ????INVA… │ ????INVA… │ ????INVA… │
│ ????INVA… │ ????INVA… │ ????INVA… │
│ ????INVA… │ ????INVA… │ ????INVA… │
│ ????INVA… │ ????INVA… │ ????INVA… │
│ ????INVA… │ ????INVA… │ ????INVA… │
│ ????INVA… │ ????INVA… │ ????INVA… │
├────────────┴────────────┴────────────┤
│ 10 rows 3 columns │
└──────────────────────────────────────┘
```

Also, `COPY TO` fails if we specify the option.

```sql
COPY (
SELECT ST_Point(1, 1), 'あああ' as text
) TO 'tmp'
WITH (FORMAT GDAL, DRIVER 'ESRI Shapefile', LAYER_CREATION_OPTIONS 'ENCODING=CP932');
```
```
IO Error:
GDAL Error (1): Failed to create field name 'text': cannot convert to CP932
```

Note that, `COPY TO`'s case isn't serious as we rarely output non-UTF-8 files. However, I really wish `ST_Read` works because, unfortunately, there are a lot of non-UTF-8 geospatial data.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.