futureverse / futureverse/progressr

Delayed progress updates to file due to file-system buffering [Was: No near-live progress update when running on a SLURM cluster]

Open
#149 17 comments 0 reactions 0 assignees View on GitHub
documentation
Dominant language
R
Stars
299
Forks
11
PR merge metrics
No merged PRs in 30d

Description

I want to run a loop in parallel on a SLURM cluster and use `progressr` to output near-live updates. Here's my setup:

### test.R
```r
library("future")
library("furrr")
library("progressr")

options(progressr.enable = TRUE)
handlers(global = TRUE)
handlers(handler_progress(
format = ":current/:total [:bar] :percent in :elapsed, ETA: :eta",
width = 60,
complete = "="
))
plan(cluster)

f <- function(i) {
Sys.sleep(3)
i
}

local({
n <- 100
p <- progressor(n)
result <- future_map(1:n, function(i){
p(class = "sticky")
f(i)
})
})
```

### test.sh
```bash
#!/bin/bash
#SBATCH --time=0-00:05:00
#SBATCH --ntasks 10
#SBATCH --output=slurm.out
Rscript test.R
```

The issue is that when the job is running, `cat slurm.out` always shows
`0/100 [--------------------------------------------------] 0% in 0s, ETA: ?` and no live progress is reported. How can I get it to show the current progress?

I have read [this](https://github.com/HenrikBengtsson/future/discussions/542) discussion on getting a progress bar using `future.batchtools` and [that](https://github.com/HenrikBengtsson/future/discussions/572) thread on printing messages to the console when they are generated, but I can't get it to work. Is this a bug?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.