mapbox / mapbox/tilelive

Memory and recursion issues in tilelive-copy with --parts

Open
#170 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
JavaScript
Stars
540
Forks
106
PR merge metrics
No merged PRs in 30d

Description

Using `tilelive-copy` with `--parts` on a large copy seems to have a few issues. In both cases, I was using:
`tilelive-copy --concurrency=24 --withoutprogress --timeout=900000 --minzoom=0 --maxzoom=12 --retry=1000 --parts=$PARTS --part=$n mapnik://./project.xml .../foo.mbtiles`

Initially, I tried `PARTS=1000`, and got a `Maximum call stack size exceeded`. The stack (now gone, sorry) looked to involve a massive recursion on `skip()/get()`, right about here:
https://github.com/mapbox/tilelive/blob/efdfe7d2b121ccc8ace74844088913f1f981abcc/lib/stream-scanline.js#L120

I figured even with the recursion, that I'd avoid blowing the stack with `PARTS=20`, and I did. But each part ended up getting killed or dying due to out of memory. The same command without `--parts` at all runs just fine (no out of memory errors).

I did not try the `pyramid` or `list` scheme, so this may or may not be limited to `scanline`.

Here are some of the versions in use:

```
| +-- tilelive@5.12.2
| | +-- minimist@0.2.0
| | `-- progress-stream@0.5.0
| | +-- single-line-log@0.3.1
| | +-- speedometer@0.1.4
| | `-- through2@0.2.3
| | +-- readable-stream@1.1.14
| | | `-- isarray@0.0.1
| | `-- xtend@2.1.2
| | `-- object-keys@0.4.0
| +-- tilelive-cache@0.6.5
| | +-- locking-cache@0.2.1
| | `-- tilelive-streaming@0.6.4
| | `-- async@0.9.2
| `-- tilelive-modules@0.2.1
```

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the reported tilelive-copy command with --parts, first comparing PARTS=1000 and PARTS=20 against a run without --parts. Start at lib/stream-scanline.js around line 120 and trace the skip()/get() recursion and memory behavior; done means large partitioned copies no longer overflow the stack or run out of memory.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript
Domain
cli
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.