duckdb / duckdb/duckdb-spatial
ST_Read is much slower than geopandas.read_file to read remote GeoJSON
- Dominant language
- C
- Stars
- 708
- Forks
- 96
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 5
Description
Compare the following:
**GeoPandas**: (4.7 seconds)
```
❯ time python3 -c 'import geopandas as gpd; print(gpd.read_file("https://services5.arcgis.com/OvOcYIrJnM97ABBA/ArcGIS/rest/services/Primary_Health_Network_2023/FeatureServer/0/query?where=PHN_CODE=%27PHN108%27&outFields=*&returnGeometry=true&f=geojson"))'
OBJECTID_1 PHN_NAME PHN_CODE ... Shape__Area Shape__Length geometry
0 1 Hunter New England and Central Coast PHN108 ... 12.552359 38.050403 MULTIPOLYGON (((152.36517 -28.36615, 152.36714...
[1 rows x 15 columns]
python3 -c 1.29s user 0.78s system 44% cpu 4.662 total
```
**ST_Read**: (12.8 seconds)
```
❯ time duckdb -c 'load httpfs; load spatial; SELECT * FROM ST_Read("https://services5.arcgis.com/OvOcYIrJnM97ABBA/ArcGIS/rest/services/Primary_Health_Network_2023/FeatureServer/0/query?where=PHN_CODE=%27PHN108%27&outFields=*&returnGeometry=true&f=geojson");'
100% ▕████████████████████████████████████████████████████████████▏
┌────────────┬──────────────────────┬──────────┬─────────────────┬───┬──────────────────────┬──────────────────┬──────────────────┬──────────────────────┐
│ OBJECTID_1 │ PHN_NAME │ PHN_CODE │ STE_NAME │ … │ WEBSITE │ Shape__Area │ Shape__Length │ geom │
│ int32 │ varchar │ varchar │ varchar │ │ varchar │ double │ double │ geometry │
├────────────┼──────────────────────┼──────────┼─────────────────┼───┼──────────────────────┼──────────────────┼──────────────────┼──────────────────────┤
│ 1 │ Hunter New England… │ PHN108 │ New South Wales │ … │ https://thephn.com… │ 12.5523589729341 │ 38.0504026046165 │ MULTIPOLYGON (((15… │
├────────────┴──────────────────────┴──────────┴─────────────────┴───┴──────────────────────┴──────────────────┴──────────────────┴──────────────────────┤
│ 1 rows 15 columns (8 shown) │
└────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
duckdb -c 1.05s user 0.18s system 9% cpu 12.842 total
```
**Requests**: (1.7 seconds)
To eliminate the network connection time, this is the same URL just downloaded with `requests`:
```
❯ time python3 -c 'import requests; print(requests.get("https://services5.arcgis.com/OvOcYIrJnM97ABBA/ArcGIS/rest/services/Primary_Health_Network_2023/FeatureServer/0/query?where=PHN_CODE=%27PHN108%27&outFields=*&returnGeometry=true&f=geojson").content)'
...
python3 -c 0.17s user 0.07s system 14% cpu 1.723 total
```
## Now with more data
If I repeat the above, but fetch all features rather than just a single feature, the timing is as follows:
**GeoPandas**: (75 seconds)
```
❯ time python3 -c 'import geopandas as gpd; print(gpd.read_file("https://services5.arcgis.com/OvOcYIrJnM97ABBA/ArcGIS/rest/services/Primary_Health_Network_2023/FeatureServer/0/query?where=1=1&outFields=*&returnGeometry=true&f=geojson"))'
OBJECTID_1 PHN_NAME PHN_CODE ... Shape__Area Shape__Length geometry
0 1 Hunter New England and Central Coast PHN108 ... 12.552359 38.050403 MULTIPOLYGON (((152.36517 -28.36615, 152.36714...
1 2 Murrumbidgee PHN110 ... 12.323726 33.699552 POLYGON ((144.13059 -35.63596, 144.13048 -35.6...
2 3 Western NSW PHN107 ... 41.926476 49.174940 POLYGON ((143.17615 -28.99903, 143.22612 -28.9...
3 4 Western Sydney PHN103 ... 0.075297 2.187267 POLYGON ((150.98395 -33.38793, 150.98405 -33.3...
4 5 North Coast PHN109 ... 3.057769 15.786198 MULTIPOLYGON (((153.47744 -28.15693, 153.47876...
5 6 South Western Sydney PHN105 ... 0.610836 5.788375 POLYGON ((150.41818 -33.80964, 150.41830 -33.8...
6 7 Nepean Blue Mountains PHN104 ... 0.889310 7.026907 POLYGON ((150.11620 -32.88529, 150.11625 -32.8...
7 8 South Eastern NSW PHN106 ... 5.003578 24.196245 MULTIPOLYGON (((149.22437 -35.34177, 149.22485...
8 9 Eastern Melbourne PHN202 ... 0.399882 4.512614 POLYGON ((145.10200 -37.26295, 145.10203 -37.2...
9 10 Gippsland PHN204 ... 4.247146 22.860895 MULTIPOLYGON (((148.13270 -36.75009, 148.13269...
10 11 South Eastern Melbourne PHN203 ... 0.297586 5.339800 MULTIPOLYGON (((144.92480 -37.81998, 144.92514...
11 12 North Western Melbourne PHN201 ... 0.324576 4.335340 POLYGON ((144.69948 -37.17629, 144.70185 -37.1...
12 13 Western Victoria PHN206 ... 8.045615 18.577138 MULTIPOLYGON (((142.28942 -35.29086, 142.29131...
13 14 Brisbane South PHN302 ... 0.354350 8.432327 MULTIPOLYGON (((153.20219 -27.35289, 153.19202...
14 15 Darling Downs and West Moreton PHN304 ... 8.939946 23.136746 POLYGON ((150.24503 -25.40720, 150.24927 -25.4...
15 16 Northern Queensland PHN307 ... 43.669852 91.239501 MULTIPOLYGON (((142.53689 -10.70158, 142.53719...
16 17 Western Queensland PHN305 ... 84.728917 70.973433 MULTIPOLYGON (((142.19266 -15.34762, 142.24007...
17 18 Central Queensland, Wide Bay, Sunshine Coast PHN306 ... 14.068116 54.227750 MULTIPOLYGON (((152.93627 -25.74359, 152.93630...
18 19 Gold Coast PHN303 ... 0.169305 4.922679 MULTIPOLYGON (((153.29273 -27.69578, 153.29268...
19 20 Brisbane North PHN301 ... 0.368287 6.835961 MULTIPOLYGON (((152.43616 -26.46019, 152.44074...
20 21 Country SA PHN402 ... 92.065362 82.466509 MULTIPOLYGON (((140.99939 -25.99639, 140.99939...
21 22 Adelaide PHN401 ... 0.154304 3.426867 MULTIPOLYGON (((138.67110 -34.61602, 138.67114...
22 23 Country WA PHN503 ... 226.849274 221.912474 MULTIPOLYGON (((126.96109 -13.74078, 126.96213...
23 24 Perth North PHN501 ... 0.288434 3.586229 POLYGON ((115.67010 -31.45513, 115.67080 -31.4...
24 25 Perth South PHN502 ... 0.497991 8.230795 MULTIPOLYGON (((115.94836 -31.92322, 115.94854...
25 26 Tasmania PHN601 ... 7.384687 77.937979 MULTIPOLYGON (((144.72712 -40.64918, 144.72739...
26 27 Australian Capital Territory PHN801 ... 0.234227 3.032133 POLYGON ((149.22437 -35.34177, 149.22157 -35.3...
27 28 Northern Territory PHN701 ... 116.223560 120.605220 MULTIPOLYGON (((132.33613 -11.11985, 132.34040...
28 29 Murray PHN205 ... 9.705086 36.629858 POLYGON ((144.61076 -36.03298, 144.61109 -36.0...
29 30 Central and Eastern Sydney PHN101 ... 0.061763 5.169171 MULTIPOLYGON (((150.99815 -33.98265, 150.99917...
30 31 Northern Sydney PHN102 ... 0.087482 5.192650 MULTIPOLYGON (((150.99299 -33.38720, 150.99306...
[31 rows x 15 columns]
python3 -c 11.84s user 1.53s system 17% cpu 1:15.20 total
```
**ST_Read**: (killed after 724 seconds)
```
❯ time duckdb -c 'load httpfs; load spatial; SELECT * FROM ST_Read("https://services5.arcgis.com/OvOcYIrJnM97ABBA/ArcGIS/rest/services/Primary_Health_Network_2023/FeatureServer/0/query?where=1=1&outFields=*&returnGeometry=true&f=geojson");'
```
I killed this after 724 seconds.
**Requests**: (11.5 seconds)
Note, I skipped printing the output here as it's too large.
```
❯ time python3 -c 'import requests; requests.get("https://services5.arcgis.com/OvOcYIrJnM97ABBA/ArcGIS/rest/services/Primary_Health_Network_2023/FeatureServer/0/query?where=1=1&outFields=*&returnGeometry=true&f=geojson")'
python3 -c 0.77s user 0.18s system 8% cpu 11.499 total
```
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.