futureverse / futureverse/future

Slow down parallel processing using future + lidar processing (lidR R package)

Open
#836 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
R
Stars
1k
Forks
92
PR merge metrics
No merged PRs in 30d

Description

Hello Future !

We have been experimenting parallel processes slow down in the use of your package with lidar processing (airborne lidar and lidR R package).

The issue is also accessible on the lidR github page: https://github.com/r-lidar/lidR/issues/835

Basically the idea is to use future to parallel point cloud processing tasks.
It is very useful to perform complete point cloud processing workflow on large datasets and for a large community.

For example, we often use following codes:
```
future::plan(future::multisession, workers = params_general$n_cores)

cat("lidR::rasterize_terrain()")
dtm <- rasterize_terrain(ctg, res = resolution, algorithm = tin())
dtm_tiles <- list.files(file.path(params_general$path_data, name_raster), pattern = "_dtm_nooverhangs\\.tif$", full.names = TRUE)
dtm = terra::vrt(dtm_tiles)
terra::writeRaster(dtm,
filename = file.path(path_products, paste0(name_raster, ".tif")),
overwrite = TRUE)

future::plan(future::sequential)
```

The code uses both future and lidR to create a raster of the terrain topography, tile-by-tile and in parallel thanks to future.

At first, the right number of cores is used (n_cores, checked using htop) but with time, this number tends to decrease and reaching the end it is very low (sometimes only 1 core used)

The issue seems to come from future since the lidR code has not changed for a lot of time.

Do you already know about this limitation ?

Contributor guide

Open the contributing guide

Research direction

No repository files or tests are mentioned. Start by reproducing the provided future::plan(future::multisession) and lidR rasterize_terrain() workflow while monitoring worker usage; done means identifying whether the slowdown is caused by future and documenting or addressing the confirmed limitation.

Written by the indexing model from the issue text.

Assessment

Tech stack
r
Domain
distributed-systems, hpc, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.