apache / apache/arrow

[C++][R] Cannot open S3 paths that are already have URL-encoding (literal percent-encoding) characters

Open
#35,714 3 comments 0 reactions 0 assignees View on GitHub
Component: C++ Component: R Priority: Critical Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

This works:

```r
library(arrow)
library(dplyr)
open_dataset("s3://neon4cast-forecasts/parquet/terrestrial_30min/model_id=climatology/reference_datetime=2023-02-15%2000:00:00?endpoint_override=data.ecoforecast.org") |> head() |> collect()
```

```r
open_dataset("s3://neon4cast-forecasts/parquet/terrestrial_30min/model_id=climatology/reference_datetime=2023-02-16%2000%3A00%3A00?endpoint_override=data.ecoforecast.org") |> head() |> collect()
```

We can see the file does indeed appear under `ls()` though:

```r
s3 <- s3_bucket("s3://neon4cast-forecasts/parquet/terrestrial_30min/model_id=climatology?endpoint_override=data.ecoforecast.org")
s3$ls()
```

We just can't access any of those dates after 2023-02-15 which use the percent encoding in the filenames by URIs. Interestingly, opening at the model root works, and is able to read from both partitions (note the filter to grab the troublesome URLescaped partition)

```r
open_dataset("s3://neon4cast-forecasts/parquet/terrestrial_30min/model_id=climatology?endpoint_override=data.ecoforecast.org") |> filter(reference_datetime=="2023-02-16 00:00:00") |> collect()
```

### Component(s)

R

Contributor guide

Open the contributing guide

Research direction

Reproduce the failure with the R examples using open_dataset() and s3_bucket()$ls() against the encoded S3 partition paths. Trace how the R S3 dataset access handles literal percent-encoded characters, then verify that both encoded date partitions can be opened and collected without changing the working root-level behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, cpp, r
Domain
cloud, data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.