apache / apache/arrow

[C++][R] copy_files() does not take paths to specific files

Open
#20,324 3 comments 0 reactions 0 assignees View on GitHub
Component: C++ Component: R Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

`copy_files()` is a pretty handy function for working between local and remote interfaces, particularly for any file type arrow doesn't handle (arvo, ncdf, h5, etc etc). 

Unfortunately, it seems to work only from directory-to-directory, at least in the direction of copying S3 -> local file system.  e.g. this reprex:

```java

library(arrow)
local_dir <- tempfile()
fs::dir_delete(local_dir)
fs::dir_create(local_dir) # create dir if it doesn't exist
l3 <- SubTreeFileSystem$create(local_dir)
l3$ls() #empty

s3 <- s3_bucket("neon4cast-targets/aquatics", endpoint_override = "data.ecoforecast.org", anonymous=TRUE)
s3$ls() #not empty, good

copy_files(s3$path("aquatics-targets.csv.gz"), l3$path("aquatics-targets.csv.gz"))
l3$ls() # darn, nothing!

copy_files(s3$path("aquatics-targets.csv.gz"), l3)
l3$ls() # darn, nothing!

copy_files(s3, l3)
l3$ls()  # Finally! only this works
```

**Reporter**: [Carl Boettiger](https://issues.apache.org/jira/browse/ARROW-17120) / @cboettig
#### Related issues:
- [[R] copy_files doesn't make it easy to copy a single file](https://github.com/apache/arrow/issues/28030) (is duplicated by)

**Note**: *This issue was originally created as [ARROW-17120](https://issues.apache.org/jira/browse/ARROW-17120). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

The report provides an R reprex using copy_files(), SubTreeFileSystem$create(), and s3_bucket(); begin by reproducing the S3-to-local single-file cases. Done means a source and destination path for one file copies it and the result appears in l3$ls(), while directory-to-directory behavior remains intact.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, cpp, r
Domain
backend, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.