openstreetmap / openstreetmap/mod_tile

render_list multi threading is not working, it seems

Open
#405 13 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
318
Forks
199
PR merge metrics
No merged PRs in 30d

Description

Hi Team,

Its rather asking for help, presumably not a bug (not sure).

I have followed this page to configure postgres with OSM planet data and configured mod_tile and renderd.

I have a cloud VM with 350GB RAM and 46 threads to use for this exercise. Hence, I have updated the postgres configure parameters as mentioned below to take most out of available resources.

shared_buffers = 200GB
work_mem = 10GB
maintenance_work_mem = 40GB
effective_cache_size = 40GB
max_connections = 100

I was able to render the tiles successfully, however, my goal is to pre-generate all zoom level tiles and hence executed below command

render_list -m default -a -z 0 -Z 19 --num-threads=40

Till zoom level 13, it was too fast but later started getting slow, I know the number of tiles to be generated gets exponential as zoom level goes up.

So tried with just zoom level only 15 with setup, surprisingly, no major consumption of RAM and the process is too slow even with 40 threads.

render_list -m default -a -z 15 -Z 15 --num-threads=40

Just wondering if this threads usage is working fine or am I missing anything to make most out of available resources to generate tiles quickly? Any suggestions would be greatly appreciated.

-Prem

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the render_list command using --num-threads=40 at zoom 15, then review the renderd configuration and PostgreSQL settings described in the report. Compare observed thread and resource usage with the expected behavior. Done would mean determining whether multithreading is functioning and documenting a concrete configuration or implementation change if it is not.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, postgresql
Domain
backend, databases, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.