Memory and recursion issues in tilelive-copy with --parts
Nobody has claimed this yet.
- Dominant language
- JavaScript
- Stars
- 540
- Forks
- 106
- PR merge metrics
- No merged PRs in 30d
Description
Using `tilelive-copy` with `--parts` on a large copy seems to have a few issues. In both cases, I was using:
`tilelive-copy --concurrency=24 --withoutprogress --timeout=900000 --minzoom=0 --maxzoom=12 --retry=1000 --parts=$PARTS --part=$n mapnik://./project.xml .../foo.mbtiles`
Initially, I tried `PARTS=1000`, and got a `Maximum call stack size exceeded`. The stack (now gone, sorry) looked to involve a massive recursion on `skip()/get()`, right about here:
https://github.com/mapbox/tilelive/blob/efdfe7d2b121ccc8ace74844088913f1f981abcc/lib/stream-scanline.js#L120
I figured even with the recursion, that I'd avoid blowing the stack with `PARTS=20`, and I did. But each part ended up getting killed or dying due to out of memory. The same command without `--parts` at all runs just fine (no out of memory errors).
I did not try the `pyramid` or `list` scheme, so this may or may not be limited to `scanline`.
Here are some of the versions in use:
```
| +-- tilelive@5.12.2
| | +-- minimist@0.2.0
| | `-- progress-stream@0.5.0
| | +-- single-line-log@0.3.1
| | +-- speedometer@0.1.4
| | `-- through2@0.2.3
| | +-- readable-stream@1.1.14
| | | `-- isarray@0.0.1
| | `-- xtend@2.1.2
| | `-- object-keys@0.4.0
| +-- tilelive-cache@0.6.5
| | +-- locking-cache@0.2.1
| | `-- tilelive-streaming@0.6.4
| | `-- async@0.9.2
| `-- tilelive-modules@0.2.1
```
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the reported tilelive-copy command with --parts, first comparing PARTS=1000 and PARTS=20 against a run without --parts. Start at lib/stream-scanline.js around line 120 and trace the skip()/get() recursion and memory behavior; done means large partitioned copies no longer overflow the stack or run out of memory.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript
- Domain
- cli
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100