mapbox / mapbox/tippecanoe

RAM exhaustion with 59,067,585-line, 102 GB JSONL file

Open
#962 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
3.1k
Forks
430
PR merge metrics
No merged PRs in 30d

Description

I'm running the following on a system with 64 cores and 64 GB of RAM. After a few hours of running the application appears to exhaust all available memory and is terminated by the Kernel.

Is there any workaround for this?

```bash
$ tippecanoe \
--coalesce-densest-as-needed \
-zg \
--extend-zooms-if-still-dropping \
-e fcc_477 \
out.geojson
```

```
For layer 0, using name "out"
59067585 features, 15368550538 bytes of geometry, 5954962706 bytes of separate metadata, 750974459 bytes of string pool
Choosing a maxzoom of -z8 for features about 1021 feet (311 meters) apart
Choosing a maxzoom of -z11 for resolution of about 138 feet (42 meters) within features
tile 0/0/0 size is 3193098 with detail 12, >500000
Going to try keeping the sparsest 14.09% of the features to make it fit
tile 0/0/0 size is 3105152 with detail 12, >500000
Going to try keeping the sparsest 2.04% of the features to make it fit
tile 0/0/0 size is 3009947 with detail 12, >500000
Going to try keeping the sparsest 0.31% of the features to make it fit
tile 0/0/0 size is 2295077 with detail 12, >500000
Going to try keeping the sparsest 0.06% of the features to make it fit
tile 0/0/0 size is 1182125 with detail 12, >500000
Going to try keeping the sparsest 0.02% of the features to make it fit
tile 0/0/0 size is 694395 with detail 12, >500000
Going to try keeping the sparsest 0.01% of the features to make it fit
tile 0/0/0 size is 544355 with detail 12, >500000
Going to try keeping the sparsest 0.01% of the features to make it fit
tile 1/0/0 has 314800 features, >200000
Going to try keeping the sparsest 57.18% of the features to make it fit
tile 1/0/0 has 272910 features, >200000
Going to try keeping the sparsest 37.71% of the features to make it fit
tile 1/0/0 has 272910 features, >200000
Going to try keeping the sparsest 24.87% of the features to make it fit
tile 1/0/0 has 272891 features, >200000
Going to try keeping the sparsest 16.41% of the features to make it fit
tile 1/0/0 has 272612 features, >200000
Going to try keeping the sparsest 10.83% of the features to make it fit
tile 1/0/0 has 271776 features, >200000
Going to try keeping the sparsest 7.17% of the features to make it fit
tile 1/0/0 has 269162 features, >200000
Going to try keeping the sparsest 4.80% of the features to make it fit
tile 1/0/0 has 263068 features, >200000
Going to try keeping the sparsest 3.28% of the features to make it fit
tile 1/0/0 has 253891 features, >200000
Going to try keeping the sparsest 2.33% of the features to make it fit
tile 1/0/0 has 240504 features, >200000
Going to try keeping the sparsest 1.74% of the features to make it fit
tile 1/0/0 has 225503 features, >200000
Going to try keeping the sparsest 1.39% of the features to make it fit
tile 1/0/0 has 212709 features, >200000
Going to try keeping the sparsest 1.18% of the features to make it fit
tile 1/0/0 has 201649 features, >200000
Going to try keeping the sparsest 1.05% of the features to make it fit
tile 1/0/0 size is 7764025 with detail 12, >500000
Going to try keeping the sparsest 0.06% of the features to make it fit
tile 1/0/0 size is 1993600 with detail 12, >500000
Going to try keeping the sparsest 0.01% of the features to make it fit
tile 1/0/0 size is 993399 with detail 12, >500000
Going to try keeping the sparsest 0.01% of the features to make it fit
tile 1/0/0 size is 771458 with detail 12, >500000
Going to try keeping the sparsest 0.00% of the features to make it fit
tile 1/0/0 size is 686409 with detail 12, >500000
Going to try keeping the sparsest 0.00% of the features to make it fit
tile 1/0/0 size is 645387 with detail 12, >500000
Going to try keeping the sparsest 0.00% of the features to make it fit
tile 1/0/0 size is 619394 with detail 12, >500000
Going to try keeping the sparsest 0.00% of the features to make it fit
tile 1/0/0 size is 600238 with detail 12, >500000
Going to try keeping the sparsest 0.00% of the features to make it fit
tile 1/0/0 size is 587567 with detail 12, >500000
Going to try keeping the sparsest 0.00% of the features to make it fit
Killed% 1/0/0
```

Here is an example record from the 102 GB JSONL file:

```bash
$ head -n1 out.geojson | jq -S .
```

```
{
"features": {
"geometry": {
"coordinates": [
[
[
-112.081989,
33.491495
],
[
-112.080484,
33.491484
],
[
-112.078974,
33.49147
],
[
-112.078186,
33.491464
],
[
-112.078179,
33.492564
],
[
-112.07817,
33.493845
],
[
-112.07817,
33.494028
],
[
-112.078171,
33.494847
],
[
-112.078538,
33.494846
],
[
-112.079147,
33.494853
],
[
-112.08012,
33.494852
],
[
-112.080399,
33.494852
],
[
-112.081093,
33.494861
],
[
-112.081441,
33.49487
],
[
-112.081442,
33.494093
],
[
-112.081443,
33.492424
],
[
-112.081452,
33.492396
],
[
-112.081472,
33.492373
],
[
-112.081501,
33.492361
],
[
-112.081869,
33.492311
],
[
-112.082009,
33.49228
],
[
-112.081989,
33.491495
]
]
],
"type": "Polygon"
},
"properties": {
"blockcode": "040131105012000",
"business": "1",
"consumer": "0",
"dba_name": "MCI",
"frn": "0010856284",
"h3_7": "8729b6d06ffffff",
"h3_8": "8829b6d061fffff",
"h3_9": "8929b6d060fffff",
"hoco_final": "Verizon Communications Inc.",
"hoco_num": "131425",
"holding_company_name": "Verizon Communications Inc.",
"log_rec_num": "4737332",
"max_ad_down": 0,
"max_ad_up": 0,
"provider_id": "76029",
"provider_name": "Verizon Business Global LLC dba Verizon Business",
"source": "fbd_us_without_satellite_jun2021_v1.csv",
"state_abbr": "AZ",
"tech_code": "30"
},
"type": "Feature"
},
"type": "FeatureCollection"
}
```

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the reported tippecanoe command with the 102 GB JSONL/GeoJSON input and the stated 64 GB RAM limit. Investigate the processing stage shown in the log where tiles are repeatedly reduced, and define done as processing this workload without exhausting memory or being killed by the kernel.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
cli, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.