futureverse / futureverse/future
Slow down parallel processing using future + lidar processing (lidR R package)
- Dominant language
- R
- Stars
- 1k
- Forks
- 92
- PR merge metrics
- No merged PRs in 30d
Description
Hello Future !
We have been experimenting parallel processes slow down in the use of your package with lidar processing (airborne lidar and lidR R package).
The issue is also accessible on the lidR github page: https://github.com/r-lidar/lidR/issues/835
Basically the idea is to use future to parallel point cloud processing tasks.
It is very useful to perform complete point cloud processing workflow on large datasets and for a large community.
For example, we often use following codes:
```
future::plan(future::multisession, workers = params_general$n_cores)
cat("lidR::rasterize_terrain()")
dtm <- rasterize_terrain(ctg, res = resolution, algorithm = tin())
dtm_tiles <- list.files(file.path(params_general$path_data, name_raster), pattern = "_dtm_nooverhangs\\.tif$", full.names = TRUE)
dtm = terra::vrt(dtm_tiles)
terra::writeRaster(dtm,
filename = file.path(path_products, paste0(name_raster, ".tif")),
overwrite = TRUE)
future::plan(future::sequential)
```
The code uses both future and lidR to create a raster of the terrain topography, tile-by-tile and in parallel thanks to future.
At first, the right number of cores is used (n_cores, checked using htop) but with time, this number tends to decrease and reaching the end it is very low (sometimes only 1 core used)
The issue seems to come from future since the lidR code has not changed for a lot of time.
Do you already know about this limitation ?
Contributor guide
Research direction
No repository files or tests are mentioned. Start by reproducing the provided future::plan(future::multisession) and lidR rasterize_terrain() workflow while monitoring worker usage; done means identifying whether the slowdown is caused by future and documenting or addressing the confirmed limitation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- r
- Domain
- distributed-systems, hpc, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100