duckdb / duckdb/duckdb-spatial
ST_READ very slow over network
- Dominant language
- C
- Stars
- 708
- Forks
- 96
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 5
Description
TLDR;
* `ST_Read` took 216s of download time (download 184Mo) to create a table from a 100Mo gpkg file.
* `wget` download it in 1.4s
* R `sf::st_read` took 4.5s to download it
I'm on a 1Gb internet link
# duckdb spatial time
I use duckdb cli 1.2.1 on linux and using ST_read over network is really slow :
```
D EXPLAIN ANALYZE CREATE OR REPLACE TABLE iris2 AS FROM ST_READ('https://creacartes.s3.eu-west-3.amazonaws.com/2024/franceentiere/iris_franceentiere_2024.gpkg');
100% ▕████████████████████████████████████████████████████████████▏
┌─────────────────────────────────────┐
│┌───────────────────────────────────┐│
││ Query Profiling Information ││
│└───────────────────────────────────┘│
└─────────────────────────────────────┘
EXPLAIN ANALYZE CREATE OR REPLACE TABLE iris2 AS FROM ST_READ('https://creacartes.s3.eu-west-3.amazonaws.com/2024/franceentiere/iris_franceentiere_2024.gpkg');
┌─────────────────────────────────────┐
│┌───────────────────────────────────┐│
││ HTTPFS HTTP Stats ││
││ ││
││ in: 184.1 MiB ││
││ out: 0 bytes ││
││ #HEAD: 107 ││
││ #GET: 6317 ││
││ #PUT: 0 ││
││ #POST: 0 ││
│└───────────────────────────────────┘│
└─────────────────────────────────────┘
┌────────────────────────────────────────────────┐
│┌──────────────────────────────────────────────┐│
││ Total Time: 216.90s ││
│└──────────────────────────────────────────────┘│
└────────────────────────────────────────────────┘
```
# wget time
Downloading directly with wget is really fast :
```
time wget -q https://creacartes.s3.eu-west-3.amazonaws.com/2024/franceentiere/iris_franceentiere_2024.gpkg
wget -q 0,04s user 0,15s system 17% cpu 1,140 total
```
# sf::st_read time
I tried sf package for comparison and it's quite fast too :
``` r
tictoc::tic()
sf::st_read('https://creacartes.s3.eu-west-3.amazonaws.com/2024/franceentiere/iris_franceentiere_2024.gpkg')
#> Reading layer `iris_franceentiere_2024' from data source
#> `https://creacartes.s3.eu-west-3.amazonaws.com/2024/franceentiere/iris_franceentiere_2024.gpkg'
#> using driver `GPKG'
#> Simple feature collection with 16383 features and 3 fields
#> Geometry type: MULTIPOLYGON
#> Dimension: XY
#> Bounding box: xmin: -63.15332 ymin: -21.38963 xmax: 55.83665 ymax: 51.0668
#> Geodetic CRS: WGS 84
tictoc::toc()
#> 4.921 sec elapsed
```
Created on 2025-04-19 with [reprex v2.1.1](https://reprex.tidyverse.org)
## is httpfs the culprit ?
I tried httpfs on a parquet file and it's normally fast
```
EXPLAIN ANALYZE CREATE TABLE test2 AS FROM read_parquet('https://static.data.gouv.fr/resources/recensement-de-la-population-fichiers-detail-logements-ordinaires-en-2020-1/20231023-123618/fd-logemt-2020.parquet');
┌─────────────────────────────────────┐
│┌───────────────────────────────────┐│
││ HTTPFS HTTP Stats ││
││ ││
││ in: 481.0 MiB ││
││ out: 0 bytes ││
││ #HEAD: 1 ││
││ #GET: 27 ││
││ #PUT: 0 ││
││ #POST: 0 ││
│└───────────────────────────────────┘│
└─────────────────────────────────────┘
┌────────────────────────────────────────────────┐
│┌──────────────────────────────────────────────┐│
││ Total Time: 8.69s ││
│└──────────────────────────────────────────────┘│
└────────────────────────────────────────────────┘
```
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.