rstudio / rstudio/reticulate

Using Python subprocess.Popen in Rmd Failed

Open
#412 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
R
Stars
1.8k
Forks
349
Avg merge
5h 13m
Merged PRs (30d)
4

Description

I am using Rmd and reticulate python eval function, which aims at opening a file via Python subprocess.Popen command, however, it seems sub-threading is not available. BTW, it works very well in the terminal.

my code is

import subprocess

def getFilefromHDFS(path):
  args_list = ['hadoop', 'fs', '-ls', path]
  proc = subprocess.Popen(args_list, stdout=subprocess.PIPE,stderr=subprocess.PIPE)
  proc.communicate()
  print proc.returncode
  
getFilefromHDFS("/user/hive/warehouse/ai.db/xxx/")

The error is :

OSError: [Errno 2] No such file or directory

here is my session info:

R version 3.5.1 (2018-07-02)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 16.04.5 LTS

Matrix products: default
BLAS: /usr/local/lib/R/lib/libRblas.so
LAPACK: /usr/local/lib/R/lib/libRlapack.so

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods  
[7] base     

other attached packages:
[1] sp_1.3-1           leaflet_1.1.0.9000 crosstalk_1.0.0   
[4] mlflow_0.7.0       bindrcpp_0.2.2     dplyr_0.7.8       

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.0       lattice_0.20-35  forge_0.1.0     
 [4] tidyr_0.8.2      png_0.1-7        ps_1.1.0        
 [7] utf8_1.1.4       assertthat_0.2.0 rprojroot_1.3-2 
[10] digest_0.6.16    packrat_0.4.9-3  showimage_1.0.0 
[13] mime_0.5         plyr_1.8.4       R6_2.3.0        
[16] backports_1.1.2  evaluate_0.11    httr_1.3.1      
[19] pillar_1.3.0     rlang_0.3.0.1    lazyeval_0.2.1  
[22] rematch_1.0.1    curl_3.2         rstudioapi_0.7  
[25] callr_3.0.0      Matrix_1.2-14    DT_0.5          
[28] reticulate_1.10  rmarkdown_1.10   readr_1.1.1     
[31] stringr_1.3.1    htmlwidgets_1.3  munsell_0.5.0   
[34] shiny_1.1.0      compiler_3.5.1   pingr_1.1.2     
[37] httpuv_1.4.5     pkgconfig_2.0.2  base64enc_0.1-3 
[40] htmltools_0.3.6  openssl_1.0.2    tidyselect_0.2.5
[43] tibble_1.4.2     shinytest_1.3.0  fansi_0.3.0     
[46] crayon_1.3.4     withr_2.1.2      later_0.7.4     
[49] grid_3.5.1       jsonlite_1.5     xtable_1.8-3    
[52] git2r_0.23.0     magrittr_1.5     scales_1.0.0    
[55] geohash_0.4.0    cli_1.0.0        stringi_1.2.4   
[58] debugme_1.1.0    swagger_3.9.2    webdriver_1.0.5 
[61] fs_1.2.6         promises_1.0.1   testthat_2.0.1  
[64] parsedate_1.1.3  tools_3.5.1      glue_1.3.0      
[67] purrr_0.2.5      hms_0.4.2        processx_3.2.0  
[70] yaml_2.2.0       colorspace_1.3-2 knitr_1.20      
[73] bindr_0.1.1 

need help,
Thanks

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the Rmd call through reticulate's Python eval with the shown subprocess.Popen example, then compare it with the terminal execution and the supplied session information. The issue does not name project files or tests; done means identifying the cause of the OSError and establishing a working or clearly documented subprocess behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, r
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.