ThinkR-open / ThinkR-open/squash

feat: Make the parallel rendering robust

Open
#17 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
R
Stars
9
Forks
0
PR merge metrics
No merged PRs in 30d

Description

Description

Right now the parallel rendering isn't reliable and returns error because it doesn't find the mediabag folder.

future::plan(future::multisession, workers = 8)

for (i in 1:100){
  write(file = sprintf("test%s.qmd", i),
        sprintf("---
title: 'test%s'
---

## Quarto is cool", i)
  )
}

list.files(pattern = "\\.qmd$", full.names = TRUE, recursive = TRUE) |>
  squash::compile_qmd_course(
    output_dir = tempdir(),
    output_html = "complete_course.html"
  )
ℹ {future} is using future::plan(future::multisession, workers = 8), to modify this use `future::plan()`
✖ Failed to render ./test18.qmd, cleaning and existing
✖ Failed to render ./test40.qmd, cleaning and existing
✖ Failed to render ./test45.qmd, cleaning and existing
✖ Failed to render ./test47.qmd, cleaning and existing
✖ Failed to render ./test54.qmd, cleaning and existing
✖ Failed to render ./test74.qmd, cleaning and existing
✖ Failed to render ./test75.qmd, cleaning and existing
✖ Failed to render ./test76.qmd, cleaning and existing
✖ Failed to render ./test78.qmd, cleaning and existing
✖ Failed to render ./test87.qmd, cleaning and existing
✖ Failed to render ./test95.qmd, cleaning and existing
Error in squash::compile_qmd_course(list.files(pattern = "\\.qmd$", full.names = TRUE,  : 
  Failed to render all qmd files.

Diagnostic is that the folder is created by one process, listed by the second, then deleted by the first, and unreachable by the second

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the provided future::multisession example and inspect the compile_qmd_course entry point, focusing on how the mediabag folder is created, listed, and deleted across workers. Done means the example reliably renders all QMD files in parallel without workers losing access to the mediabag folder.

Written by the indexing model from the issue text.

Assessment

Tech stack
r
Domain
build-system, tooling
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.