[R] pillar::glimpse takes too long
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the enhancement requested
**glimpse** is loading the data to memory which make it a really slow process.
For example, using the [High Volume For-Hire Vehicle Trip Records](https://www.nyc.gov/site/tlc/about/tlc-trip-record-data.page) of 2022.
```r
> library(dplyr)
> library(arrow)
> NycTrips2022 <-
+ here::here("data/trip-data/year=2022") |>
+ open_dataset()
>
> system.time(glimpse(NycTrips2022))
FileSystemDataset with 12 Parquet files
212,416,083 rows x 25 columns
$ hvfhs_license_num "HV0003", "HV0003", "HV0003", "HV0003",…
$ dispatching_base_num "B03404", "B03404", "B03404", "B03404",…
$ originating_base_num "B03404", "B03404", "B03404", "B03404",…
$ request_datetime 2021-12-31 20:05:31, 2021-12-31 20:19:2…
$ on_scene_datetime 2021-12-31 20:05:40, 2021-12-31 20:22:0…
$ pickup_datetime 2021-12-31 20:07:24, 2021-12-31 20:22:3…
$ dropoff_datetime 2021-12-31 20:18:28, 2021-12-31 20:30:1…
$ PULocationID 170, 237, 237, 262, 229, 263, 113, 151,…
$ DOLocationID 161, 161, 161, 229, 141, 79, 140, 75, 2…
$ trip_miles 1.180, 0.820, 1.180, 1.650, 1.650, 4.51…
$ trip_time 664, 460, 595, 303, 461, 762, 931, 843,…
$ base_passenger_fare 24.90, 11.97, 29.82, 7.91, 9.44, 17.67,…
$ tolls 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.0…
$ bcf 0.75, 0.36, 0.89, 0.24, 0.28, 0.53, 0.5…
$ sales_tax 2.21, 1.06, 2.65, 0.70, 0.84, 1.57, 1.4…
$ congestion_surcharge 2.75, 2.75, 2.75, 2.75, 2.75, 2.75, 2.7…
$ airport_fee 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,…
$ tips 0, 0, 0, 0, 0, 0, 0, 4, 0, 0, 0, 0, 0, …
$ driver_pay 23.03, 12.32, 23.30, 6.30, 7.44, 12.25,…
$ shared_request_flag "N", "N", "N", "N", "N", "N", "N", "N",…
$ shared_match_flag "N", "N", "N", "N", "N", "N", "N", "N",…
$ access_a_ride_flag " ", " ", " ", " ", " ", " ", " ", " ",…
$ wav_request_flag "N", "N", "N", "N", "N", "N", "N", "N",…
$ wav_match_flag "N", "N", "N", "N", "N", "N", "N", "N",…
$ month 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, …
Call `print()` for full schema details
user system elapsed
222.25 174.87 157.95
> sessionInfo()
R version 4.2.3 (2023-03-15 ucrt)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 19045)
Matrix products: default
locale:
[1] LC_COLLATE=English_United States.utf8
[2] LC_CTYPE=English_United States.utf8
[3] LC_MONETARY=English_United States.utf8
[4] LC_NUMERIC=C
[5] LC_TIME=English_United States.utf8
attached base packages:
[1] stats graphics grDevices datasets utils methods base
other attached packages:
[1] arrow_14.0.0.1 dplyr_1.1.4
loaded via a namespace (and not attached):
[1] here_1.0.1 fansi_1.0.5 rprojroot_2.0.4
[4] tzdb_0.4.0 assertthat_0.2.1 utf8_1.2.4
[7] R6_2.5.1 lifecycle_1.0.4 magrittr_2.0.3
[10] pillar_1.9.0 rlang_1.1.2 cli_3.6.1
[13] renv_1.0.3 rstudioapi_0.15.0 vctrs_0.6.4
[16] generics_0.1.3 tools_4.2.3 bit64_4.0.5
[19] glue_1.6.2 purrr_1.0.2 bit_4.0.5
[22] yaml_2.3.7 compiler_4.2.3 pkgconfig_2.0.3
[25] tidyselect_1.2.0 tibble_3.2.1
```
To solve this problem, we could create a method for **glimpse**.
### Component(s)
R
Contributor guide
Assessment
This issue has not been assessed yet.