rstudio / rstudio/packrat

packrat symlink behavior causes errors when running many R sessions simultaneously

Open
#276 21 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
R
Stars
409
Forks
89
PR merge metrics
No merged PRs in 30d

Description

Sorry, this is my last issue! (and thanks for all the help, packrat is a fantastic tool :)

I'm running simulations in R on a computer cluster, by running the same Rscript job on many nodes in parallel. However, since packrat deletes and recreates symlinks, it causes several errors across jobs (in a sporadic fashion).

I can work around this by staggering each job, so they don't begin at the same time. However, it seems like an option to disable the deletion and recreation of symlinks would bring peace of mind when turning packrat mode on! Going further, an option to disable the creation or deletion of anything, as if running packrat in "read-only" mode seems like it would be valuable for making packrat mode very simple and intuitive.

errors with base package symlinks

Error in if (nt == 1 && file.exists(to) && file.info(to)$isdir) to <- file.path(to,  : 
  missing value where TRUE/FALSE needed
Calls: source ... setPackratModeOn -> afterPackratModeOn -> symlinkSystemPackages
In addition: Warning messages:
1: In readLines(file) :
  cannot open compressed file '/jukebox/hasson/michael/dissertation/paper-pattern-similarity/packrat/lib-R/base/DESCRIPTION', probable reason 'No such file or 
directory'
2: In value[[3L]](cond) :
  Unable to read DESCRIPTION file associated with 'base' package
Execution halted

warnings with external package symlinks (I think I've gotten errors here also, but would need to start a bunch of jobs and cross my fingers)

Warning messages:
1: In file.symlink(from, to) :
  cannot symlink '/usr/people/machow/R/library/RCurl' to '/jukebox/hasson/michael/dissertation/paper-pattern-similarity/packrat/lib-ext/RCurl', reason 'File ex
ists'
2: In file.symlink(from, to) :
  cannot symlink '/usr/people/machow/R/library/bitops' to '/jukebox/hasson/michael/dissertation/paper-pattern-similarity/packrat/lib-ext/bitops/bitops', reason
 'File exists'
3: In symlinkExternalPackages(project = project) :
  The following external packages could not be linked into the packrat private library:
- 'RCurl', 'bitops'
4: In file.symlink(from, to) :
  cannot symlink '/usr/people/machow/R/library/RCurl' to '/jukebox/hasson/michael/dissertation/paper-pattern-similarity/packrat/lib-ext/RCurl/RCurl', reason 'F
ile exists'
5: In file.symlink(from, to) :
  cannot symlink '/usr/people/machow/R/library/bitops' to '/jukebox/hasson/michael/dissertation/paper-pattern-similarity/packrat/lib-ext/bitops/bitops', reason
 'File exists'
6: In symlinkExternalPackages(project = project) :
  The following external packages could not be linked into the packrat private library:
- 'RCurl', 'bitops'

I dug around a bit in the source code to figure out what was going on, and am happy to take a pass at implementing something if pointed in the right direction.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the setPackratModeOn -> afterPackratModeOn -> symlinkSystemPackages path shown in the error, then inspect symlinkExternalPackages for the external-package warnings. Done means concurrent R sessions no longer fail or warn because symlink operations race, with the requested read-only or symlink-preserving behavior defined and covered by verification.

Written by the indexing model from the issue text.

Assessment

Tech stack
r
Domain
tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.