duckdb / duckdb/duckdb-spatial

Geodesk indexed osm pbf files are not supported

Open
#641 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
C
Stars
708
Forks
96
Avg merge
1d 21h
Merged PRs (30d)
5

Description

Hey!

I tried to work with the data from https://openplanetdata.com/
```
# Takes almost 100gb of space and a while to download
$ rclone copy \
--http-url https://download.openplanetdata.com \
:http:osm/planet/pbf/planet-latest.osm.gol . \
--multi-thread-cutoff 0 \
--multi-thread-streams 64 \
--multi-thread-chunk-size 512M \
--transfers 1 --progress

# The file itself seems to be working well
$ duckdb
> DESCRIBE SELECT * FROM ST_READOSM('planet-latest.osm.gol');
┌─────────────┬──────────────────────────────────────────────┬─────────┬─────────┬─────────┬─────────┐
│ column_name │ column_type │ null │ key │ default │ extra │
│ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │
├─────────────┼──────────────────────────────────────────────┼─────────┼─────────┼─────────┼─────────┤
│ kind │ ENUM('node', 'way', 'relation', 'changeset') │ YES │ NULL │ NULL │ NULL │
│ id │ BIGINT │ YES │ NULL │ NULL │ NULL │
│ tags │ MAP(VARCHAR, VARCHAR) │ YES │ NULL │ NULL │ NULL │
│ refs │ BIGINT[] │ YES │ NULL │ NULL │ NULL │
│ lat │ DOUBLE │ YES │ NULL │ NULL │ NULL │
│ lon │ DOUBLE │ YES │ NULL │ NULL │ NULL │
│ ref_roles │ VARCHAR[] │ YES │ NULL │ NULL │ NULL │
│ ref_types │ ENUM('node', 'way', 'relation')[] │ YES │ NULL │ NULL │ NULL │
└─────────────┴──────────────────────────────────────────────┴─────────┴─────────┴─────────┴─────────┘
```

But when I try to query this large dataset it fails:
```
SELECT *
FROM ST_READOSM('planet-latest.osm.gol')
WHERE kind = 'node'
AND tags IS NOT NULL
AND lat BETWEEN 68.59 AND 68.60
AND lon BETWEEN 16.59 AND 17.00;
INTERNAL Error:
Requested allocation size of 18446744072384928378 is out of range - maximum allocation size is 281474976710656

Stack Trace:

/home/onnimonni/.duckdb/extensions/v1.2.0/linux_amd64/spatial.duckdb_extension(+0x8c55e1) [0x7f28108c55e1]
/home/onnimonni/.duckdb/extensions/v1.2.0/linux_amd64/spatial.duckdb_extension(+0x8c5644) [0x7f28108c5644]
/home/onnimonni/.duckdb/extensions/v1.2.0/linux_amd64/spatial.duckdb_extension(+0x8c7871) [0x7f28108c7871]
/home/onnimonni/.duckdb/extensions/v1.2.0/linux_amd64/spatial.duckdb_extension(+0x269662) [0x7f2810269662]
/home/onnimonni/.duckdb/extensions/v1.2.0/linux_amd64/spatial.duckdb_extension(+0x8f2dc8) [0x7f28108f2dc8]
/home/onnimonni/.duckdb/extensions/v1.2.0/linux_amd64/spatial.duckdb_extension(+0x35b786) [0x7f281035b786]
/home/onnimonni/.duckdb/extensions/v1.2.0/linux_amd64/spatial.duckdb_extension(+0x35c232) [0x7f281035c232]
duckdb() [0x167266c]
duckdb() [0x166ad9e]
duckdb() [0xcb93d6]
duckdb() [0xcb963a]
duckdb() [0xcbfb41]
duckdb() [0xcbfc22]
duckdb() [0xcb8e1a]
duckdb() [0xcb8d4c]
duckdb() [0xcc9490]
duckdb() [0xcbc829]
duckdb() [0xcc5b82]
duckdb() [0xc7e6a1]
duckdb() [0xc7e98c]
duckdb() [0xc7eb94]
duckdb() [0x7db622]
duckdb() [0x7cb559]
duckdb() [0x7cbb67]
duckdb() [0x7cc6cb]
duckdb() [0x7ccdd7]
duckdb() [0x7ac16f]
/nix/store/6q2mknq81cyscjmkv72fpcsvan56qhmg-glibc-2.40-66/lib/libc.so.6(+0x2a1fe) [0x7f297b3891fe]
/nix/store/6q2mknq81cyscjmkv72fpcsvan56qhmg-glibc-2.40-66/lib/libc.so.6(__libc_start_main+0x89) [0x7f297b3892b9]
duckdb() [0x7b14a5]

This error signals an assertion failure within DuckDB. This usually occurs due to unexpected conditions or errors in the program's logic.
For more information, see https://duckdb.org/docs/dev/internal_errors
```

**EDIT:**

I was able to use their normal pbf encoded file `planet-latest.osm.pbf` properly so it seems that this is happening because of the geodesk indexing done on the first file.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.