apache / apache/arrow

[R] Filter operations not shown when called before summarise

Open
#14,732 7 comments 0 reactions 0 assignees View on GitHub
Component: R Priority: Medium Status: needs champion Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

### Describe the bug, including details regarding any error messages, version, and platform.

When I print an arrow dataset-query that involves filtering then summarising, the filtering operation is not shown.

For example:
``` r
library(arrow)
library(dplyr)
ds_file <- file.path(tempdir(), "mtcars")

write_dataset(mtcars |> select(mpg, cyl), ds_file)
ds <- open_dataset(ds_file)

# filter is printed | EXPECTED
ds |> filter(mpg > 25)
#> FileSystemDataset (query)
#> mpg: double
#> cyl: double
#>
#> * Filter: (mpg > 25) #<======
#> See $.data for the source Arrow object

# filter is NOT printed | EXPECTED
ds |>
filter(mpg > 25) |>
summarise(mpg = mean(mpg))
#> FileSystemDataset (query)
#> mpg: double
#> #<==== Missing?!
#> See $.data for the source Arrow object

# first is NOT printed | NOT EXPECTED!
# second filter is printed | EXPECTED
ds |>
filter(mpg > 25) |>
summarise(mpg = mean(mpg)) |>
filter(mpg > 0)
#> FileSystemDataset (query)
#> mpg: double
#>
#> * Filter: (mpg > 0) #<==== Missing mpg > 25 ?!
#> See $.data for the source Arrow object
```

I would expect to see the filtering as well as the summarise command of the query as well.

I use R 4.1.1 with arrow version 10.0.0

```
Arrow package version: 10.0.0

Capabilities:

dataset TRUE
substrait FALSE
parquet TRUE
json TRUE
s3 FALSE
gcs FALSE
utf8proc TRUE
re2 TRUE
snappy TRUE
gzip FALSE
brotli FALSE
zstd FALSE
lz4 TRUE
lz4_frame TRUE
lzo FALSE
bz2 FALSE
jemalloc FALSE
mimalloc TRUE

To reinstall with more optional capabilities enabled, see
https://arrow.apache.org/docs/r/articles/install.html

Memory:

Allocator mimalloc
Current 0 bytes
Max 0 bytes

Runtime:

SIMD Level avx2
Detected SIMD Level avx2

Build:

C++ Library Version 10.0.0
C++ Compiler GNU
C++ Compiler Version 7.5.0
```

### Component(s)

R

Contributor guide

Open the contributing guide

Research direction

Run the supplied R reproducer with Arrow 10.0.0, focusing on the printed output for chained filter() and summarise() operations. Trace the dataset-query print path from this example and verify that the initial filter and summarise operation remain visible alongside a later filter.

Written by the indexing model from the issue text.

Assessment

Tech stack
r
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.