[C++][R] Cannot open S3 paths that are already have URL-encoding (literal percent-encoding) characters
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the bug, including details regarding any error messages, version, and platform.
This works:
```r
library(arrow)
library(dplyr)
open_dataset("s3://neon4cast-forecasts/parquet/terrestrial_30min/model_id=climatology/reference_datetime=2023-02-15%2000:00:00?endpoint_override=data.ecoforecast.org") |> head() |> collect()
```
```r
open_dataset("s3://neon4cast-forecasts/parquet/terrestrial_30min/model_id=climatology/reference_datetime=2023-02-16%2000%3A00%3A00?endpoint_override=data.ecoforecast.org") |> head() |> collect()
```
We can see the file does indeed appear under `ls()` though:
```r
s3 <- s3_bucket("s3://neon4cast-forecasts/parquet/terrestrial_30min/model_id=climatology?endpoint_override=data.ecoforecast.org")
s3$ls()
```
We just can't access any of those dates after 2023-02-15 which use the percent encoding in the filenames by URIs. Interestingly, opening at the model root works, and is able to read from both partitions (note the filter to grab the troublesome URLescaped partition)
```r
open_dataset("s3://neon4cast-forecasts/parquet/terrestrial_30min/model_id=climatology?endpoint_override=data.ecoforecast.org") |> filter(reference_datetime=="2023-02-16 00:00:00") |> collect()
```
### Component(s)
R
Contributor guide
Research direction
Reproduce the failure with the R examples using open_dataset() and s3_bucket()$ls() against the encoded S3 partition paths. Trace how the R S3 dataset access handles literal percent-encoded characters, then verify that both encoded date partitions can be opened and collected without changing the working root-level behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, cpp, r
- Domain
- cloud, data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100