rstudio / rstudio/packrat

packrat + dplyr results in a C stack usage error

Open
#459 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
R
Stars
409
Forks
89
No merged PRs in 30d

Description

I have boiled this down into a hopefully reproducible set of steps.

  1. Create a new Project in RStudio.
  2. Create an R file in that project, test-pr.R
  3. Add library(dplyr) to that test-pr.R file.
  4. Save, and run packrat::init()
  5. This takes awhile and crashes with the error:
# > packrat::init()
# Initializing packrat project in directory:
#   - "~/Desktop/R/test-packrat"
# Error: C stack usage  7970264 is too close to the limit

Traceback:

This shows that the recursive loop in getPackageRecords(inferredPkgsNotInLib, ...) is causing the problem. Are there just too many dependencies to recurse through? I'd be interested in upping the stack size with ulimit and retrying if you can show me how to do that from within the current R process.

...this continues up into the 200's

21: lapply(allRecords, function(record) {
        deps <- getPackageDependencies(pkgs = record$name, lib.loc = lib.loc, 
            available.packages = available)
        if (!is.null(deps)) {
            record$depends <- getPackageRecords(deps, project = project, 
                available, TRUE, lib.loc = lib.loc, missing.package = missing.package, 
                check.lockfile = check.lockfile, fallback.ok = fallback.ok)
        }
        record
    })
20: getPackageRecords(deps, project = project, available, TRUE, lib.loc = lib.loc, 
        missing.package = missing.package, check.lockfile = check.lockfile, 
        fallback.ok = fallback.ok)
19: FUN(X[[i]], ...)
18: lapply(allRecords, function(record) {
        deps <- getPackageDependencies(pkgs = record$name, lib.loc = lib.loc, 
            available.packages = available)
        if (!is.null(deps)) {
            record$depends <- getPackageRecords(deps, project = project, 
                available, TRUE, lib.loc = lib.loc, missing.package = missing.package, 
                check.lockfile = check.lockfile, fallback.ok = fallback.ok)
        }
        record
    })
17: getPackageRecords(deps, project = project, available, TRUE, lib.loc = lib.loc, 
        missing.package = missing.package, check.lockfile = check.lockfile, 
        fallback.ok = fallback.ok)
16: FUN(X[[i]], ...)
15: lapply(allRecords, function(record) {
        deps <- getPackageDependencies(pkgs = record$name, lib.loc = lib.loc, 
            available.packages = available)
        if (!is.null(deps)) {
            record$depends <- getPackageRecords(deps, project = project, 
                available, TRUE, lib.loc = lib.loc, missing.package = missing.package, 
                check.lockfile = check.lockfile, fallback.ok = fallback.ok)
        }
        record
    })
14: getPackageRecords(deps, project = project, available, TRUE, lib.loc = lib.loc, 
        missing.package = missing.package, check.lockfile = check.lockfile, 
        fallback.ok = fallback.ok)
13: FUN(X[[i]], ...)
12: lapply(allRecords, function(record) {
        deps <- getPackageDependencies(pkgs = record$name, lib.loc = lib.loc, 
            available.packages = available)
        if (!is.null(deps)) {
            record$depends <- getPackageRecords(deps, project = project, 
                available, TRUE, lib.loc = lib.loc, missing.package = missing.package, 
                check.lockfile = check.lockfile, fallback.ok = fallback.ok)
        }
        record
    })
11: getPackageRecords(deps, project = project, available, TRUE, lib.loc = lib.loc, 
        missing.package = missing.package, check.lockfile = check.lockfile, 
        fallback.ok = fallback.ok)
10: FUN(X[[i]], ...)
9: lapply(allRecords, function(record) {
       deps <- getPackageDependencies(pkgs = record$name, lib.loc = lib.loc, 
           available.packages = available)
       if (!is.null(deps)) {
           record$depends <- getPackageRecords(deps, project = project, 
               available, TRUE, lib.loc = lib.loc, missing.package = missing.package, 
               check.lockfile = check.lockfile, fallback.ok = fallback.ok)
       }
       record
   })
8: getPackageRecords(deps, project = project, available, TRUE, lib.loc = lib.loc, 
       missing.package = missing.package, check.lockfile = check.lockfile, 
       fallback.ok = fallback.ok)
7: FUN(X[[i]], ...)
6: lapply(allRecords, function(record) {
       deps <- getPackageDependencies(pkgs = record$name, lib.loc = lib.loc, 
           available.packages = available)
       if (!is.null(deps)) {
           record$depends <- getPackageRecords(deps, project = project, 
               available, TRUE, lib.loc = lib.loc, missing.package = missing.package, 
               check.lockfile = check.lockfile, fallback.ok = fallback.ok)
       }
       record
   })
5: getPackageRecords(inferredPkgsNotInLib, project = project, available = available, 
       check.lockfile = TRUE, fallback.ok = fallback.ok)
4: snapshotImpl(project, lib.loc = NULL, ignore.stale = TRUE, fallback.ok = TRUE, 
       infer.dependencies = infer.dependencies)
3: initImpl(project, options, enter, restart, infer.dependencies)
2: withCallingHandlers(initImpl(project, options, enter, restart, 
       infer.dependencies), error = function(e) {
       for (i in seq_along(priorStructure)) {
           file <- names(priorStructure)[[i]]
           fileExistedBefore <- priorStructure[[i]]
           fileExistsNow <- file.exists(file)
           if (!fileExistedBefore && fileExistsNow) {
               unlink(file, recursive = TRUE)
           }
       }
   })
1: packrat::init()

As the details below show, I am on a Mac running R 3.4.3 with CRAN dplyr and Github packrat, but CRAN packrat fails as well.

devtools::session_info()
#> Session info -------------------------------------------------------------
#>  setting  value                       
#>  version  R version 3.4.3 (2017-11-30)
#>  system   x86_64, darwin15.6.0        
#>  ui       X11                         
#>  language (EN)                        
#>  collate  en_US.UTF-8                 
#>  tz       America/New_York            
#>  date     2018-02-22
#> Packages -----------------------------------------------------------------
#>  package    * version    date      
#>  assertthat   0.2.0      2017-04-11
#>  backports    1.1.2      2017-12-13
#>  base       * 3.4.3      2017-12-07
#>  bindr        0.1        2016-11-13
#>  bindrcpp     0.2        2017-06-17
#>  compiler     3.4.3      2017-12-07
#>  datasets   * 3.4.3      2017-12-07
#>  devtools     1.13.4     2017-11-09
#>  digest       0.6.14     2018-01-14
#>  dplyr      * 0.7.4      2017-09-28
#>  evaluate     0.10.1     2017-06-24
#>  glue         1.2.0      2017-10-29
#>  graphics   * 3.4.3      2017-12-07
#>  grDevices  * 3.4.3      2017-12-07
#>  htmltools    0.3.6      2017-04-28
#>  knitr        1.17       2017-08-10
#>  magrittr     1.5        2014-11-22
#>  memoise      1.1.0      2017-04-21
#>  methods    * 3.4.3      2017-12-07
#>  packrat    * 0.4.8-57   2018-02-22
#>  pillar       1.1.0      2018-01-25
#>  pkgconfig    2.0.1      2017-03-21
#>  R6           2.2.2      2017-06-17
#>  Rcpp         0.12.15    2018-01-20
#>  rlang        0.1.6.9003 2018-02-15
#>  rmarkdown    1.6.0.9004 2017-09-23
#>  rprojroot    1.3-2      2018-01-03
#>  stats      * 3.4.3      2017-12-07
#>  stringi      1.1.6      2017-11-17
#>  stringr      1.2.0      2017-02-18
#>  tibble       1.4.2      2018-01-22
#>  tools        3.4.3      2017-12-07
#>  utils      * 3.4.3      2017-12-07
#>  withr        2.1.1.9000 2018-01-19
#>  yaml         2.1.14     2016-11-12
#>  source                                 
#>  CRAN (R 3.4.0)                         
#>  cran (@1.1.2)                          
#>  local                                  
#>  CRAN (R 3.4.0)                         
#>  CRAN (R 3.4.0)                         
#>  local                                  
#>  local                                  
#>  CRAN (R 3.4.2)                         
#>  cran (@0.6.14)                         
#>  CRAN (R 3.4.2)                         
#>  CRAN (R 3.4.0)                         
#>  CRAN (R 3.4.2)                         
#>  local                                  
#>  local                                  
#>  CRAN (R 3.4.0)                         
#>  cran (@1.17)                           
#>  CRAN (R 3.4.0)                         
#>  CRAN (R 3.4.0)                         
#>  local                                  
#>  Github (rstudio/packrat@474fea7)       
#>  Github (r-lib/pillar@872357a)          
#>  CRAN (R 3.4.0)                         
#>  CRAN (R 3.4.0)                         
#>  cran (@0.12.15)                        
#>  Github (tidyverse/rlang@a571645)       
#>  Github (DavisVaughan/rmarkdown@f85ba35)
#>  cran (@1.3-2)                          
#>  local                                  
#>  cran (@1.1.6)                          
#>  CRAN (R 3.4.0)                         
#>  cran (@1.4.2)                          
#>  local                                  
#>  local                                  
#>  Github (jimhester/withr@df18523)       
#>  CRAN (R 3.4.0)

I should also note that I found this problem while trying to push an Rmd that used a package that depended on dplyr to RStudio Connect and it failed, so that kind of sucked.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the failure with the listed RStudio, dplyr, and packrat::init steps, then inspect getPackageRecords and its callers in the traceback, including snapshotImpl and initImpl. Determine why dependency traversal recurses until the C stack limit; done means the reproducible project can complete packrat::init without the C stack usage error.

Written by the indexing model from the issue text.

Assessment

Tech stack
r
Domain
tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.