duckdb / duckdb/duckdb-spatial

Performance improvements for OSM imports?

Open
#54 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
708
Forks
96
Avg merge
1d 21h
Merged PRs (30d)
5

Description

The following OSM file is 618 MB.

```bash
$ wget https://download.geofabrik.de/australia-oceania/australia-latest.osm.pbf
```

Below are the record counts for each layer.

Layer | Record Count
------|-------------
Points | 2,659,810
Lines | 3,201,975
MultiLineStrings | 3,332
MultiPolygons | 23,308
Other Relations | 38,863

The following ran on osm2pgsql 1.8.1, PG 14.7 and PostGIS 3.3. It finished in 5m14.235s on my 2020 MBP running on an external SSD. This includes the time it spent analysing and setting up indices.

```bash
$ time osm2pgsql -d osm_aus australia-latest.osm.pbf
```

```
2023-04-18 08:38:12 Setting up table 'planet_osm_point'
2023-04-18 08:38:12 Setting up table 'planet_osm_line'
2023-04-18 08:38:12 Setting up table 'planet_osm_polygon'
2023-04-18 08:38:12 Setting up table 'planet_osm_roads'
2023-04-18 08:40:59 Reading input files done in 167s (2m 47s).
2023-04-18 08:40:59 Processed 89868343 nodes in 14s - 6419k/s
2023-04-18 08:40:59 Processed 6939120 ways in 114s (1m 54s) - 61k/s
2023-04-18 08:40:59 Processed 167603 relations in 39s - 4k/s
2023-04-18 08:41:02 Clustering table 'planet_osm_roads' by geometry...
2023-04-18 08:41:02 Clustering table 'planet_osm_line' by geometry...
2023-04-18 08:41:02 Clustering table 'planet_osm_point' by geometry...
2023-04-18 08:41:02 Clustering table 'planet_osm_polygon' by geometry...
2023-04-18 08:41:24 Creating geometry index on table 'planet_osm_roads'...
2023-04-18 08:41:28 Creating geometry index on table 'planet_osm_point'...
2023-04-18 08:41:31 Analyzing table 'planet_osm_roads'...
2023-04-18 08:42:06 Analyzing table 'planet_osm_point'...
2023-04-18 08:42:06 All postprocessing on table 'planet_osm_point' done in 64s (1m 4s).
2023-04-18 08:42:20 Creating geometry index on table 'planet_osm_polygon'...
2023-04-18 08:42:22 Creating geometry index on table 'planet_osm_line'...
2023-04-18 08:43:15 Analyzing table 'planet_osm_line'...
2023-04-18 08:43:18 All postprocessing on table 'planet_osm_line' done in 135s (2m 15s).
2023-04-18 08:43:23 Analyzing table 'planet_osm_polygon'...
2023-04-18 08:43:26 All postprocessing on table 'planet_osm_polygon' done in 143s (2m 23s).
2023-04-18 08:43:26 All postprocessing on table 'planet_osm_roads' done in 29s.
2023-04-18 08:43:26 osm2pgsql took 314s (5m 14s) overall.
```

```
osm_aus=# \d+
```

```
List of relations
Schema | Name | Type | Owner | Persistence | Access method | Size | Description
--------+--------------------+-------+-------+-------------+---------------+---------+-------------
public | geography_columns | view | mark | permanent | | 0 bytes |
public | geometry_columns | view | mark | permanent | | 0 bytes |
public | planet_osm_line | table | mark | permanent | heap | 1335 MB |
public | planet_osm_point | table | mark | permanent | heap | 210 MB |
public | planet_osm_polygon | table | mark | permanent | heap | 1458 MB |
public | planet_osm_roads | table | mark | permanent | heap | 229 MB |
public | spatial_ref_sys | table | mark | permanent | heap | 6936 kB |
```

I can't see a specific 'other relations' table but I can't rule out that data being used elsewhere. This is an example 'other relations' table.

```
osm_id = 2272
name = Causeway Bridge (Victoria Park End)
type = bridge
other_tags = "layer"=>"1","length"=>"225 m","wikidata"=>"Q54378072"
```

The following finished in a little over 9 minutes producing a 1.6 GB DuckDB file on the same system running the latest master branch compiled for release. The file is half the size of the PG DB but this is almost 2x slower that the PG toolset.

```bash
LAYERS="points lines multilinestrings multipolygons other_relations"

for LAYER in $LAYERS; do
time echo "CREATE OR REPLACE TABLE osm_$LAYER AS
SELECT *
FROM st_read('australia-latest.osm.pbf',
open_options=['INTERLEAVED_READING=YES'],
layer='$LAYER',
sequential_layer_scan=true);" \
| /Volumes/Seagate/duckdb_spatial/build/release/duckdb \
-unsigned \
aus2.duckdb
done
```

The is an import time breakdown by layer.

Layer | Duration
------|---------
points | 0m31.640s
lines | 2m59.783s
multilinestrings | 1m54.469s
multipolygons | 1m58.296s
other_relations | 1m38.887s

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.