mapbox / mapbox/tippecanoe

Tippecanoe holds forever above 5 million points in geojson file

Open
#868 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
3.1k
Forks
430
PR merge metrics
No merged PRs in 30d

Description

I have a gzipped file `positions.json.gz` with one geojson object representing one point per line:

```json
{"type":"Feature","geometry":{"type":"Point","coordinates":[-88.55666,30.10603]},"properties":{"speed":0}}
{"type":"Feature","geometry":{"type":"Point","coordinates":[-88.55661,30.10625]},"properties":{"speed":0}}
{"type":"Feature","geometry":{"type":"Point","coordinates":[-88.55695,30.106083]},"properties":{"speed":0}}
```

I run tippecanoe on Ubuntu 16.04 in parallel mode with the following command:

```shell
tippecanoe -o positions.mbtiles -P -z 22 -B 15 -r 1.25 --maximum-tile-bytes=5000000 positions.json.gz --force
```

This works great when there are fewer than approximatively 5 million rows in the file. As the count of rows increases, I noticed that tippecanoe takes more time to really start "processing" the points across multiple CPU cores. This seems normal as I'm assuming that tippecanoe simply loads the rows in memory before starting the parallel computation.

However, above 5 million rows, this "loading" step seems to hold forever and the multi-processing computation never actually starts. I've waited for several hours with 10 million rows with nothing happening.

Is this a bug or am I doing anything wrong?

*note 1: if I remove `-P` from the command to let tippecanoe run on a single core, the behaviour doesn't change: it will hold forever above 5 million points as well*

*note 2: if I don't pass any of the `-z`, `-B`, `-r` or `--maximum-tile-bytes` parameters then behaviour is still the same again*

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the reported hang with the supplied tippecanoe command and positions.json.gz, comparing inputs below and above five million rows and with and without -P. No source file or test is named in the issue; done means identifying why processing does not begin at larger inputs and verifying a fix against the reported cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
cli, data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.