futureverse / futureverse/progressr
Delayed progress updates to file due to file-system buffering [Was: No near-live progress update when running on a SLURM cluster]
- Dominant language
- R
- Stars
- 299
- Forks
- 11
- PR merge metrics
- No merged PRs in 30d
Description
I want to run a loop in parallel on a SLURM cluster and use `progressr` to output near-live updates. Here's my setup:
### test.R
```r
library("future")
library("furrr")
library("progressr")
options(progressr.enable = TRUE)
handlers(global = TRUE)
handlers(handler_progress(
format = ":current/:total [:bar] :percent in :elapsed, ETA: :eta",
width = 60,
complete = "="
))
plan(cluster)
f <- function(i) {
Sys.sleep(3)
i
}
local({
n <- 100
p <- progressor(n)
result <- future_map(1:n, function(i){
p(class = "sticky")
f(i)
})
})
```
### test.sh
```bash
#!/bin/bash
#SBATCH --time=0-00:05:00
#SBATCH --ntasks 10
#SBATCH --output=slurm.out
Rscript test.R
```
The issue is that when the job is running, `cat slurm.out` always shows
`0/100 [--------------------------------------------------] 0% in 0s, ETA: ?` and no live progress is reported. How can I get it to show the current progress?
I have read [this](https://github.com/HenrikBengtsson/future/discussions/542) discussion on getting a progress bar using `future.batchtools` and [that](https://github.com/HenrikBengtsson/future/discussions/572) thread on printing messages to the console when they are generated, but I can't get it to work. Is this a bug?
Contributor guide
Assessment
This issue has not been assessed yet.