rgee does not work with drive (edit: sometimes?!)

Open
#360 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
25/100
Issue type
Bug
Clarity
Needs clarification
Activity status
Stale
Tech stack
r
Domain
cloud, data

Research direction

Start with the ee_extract() example in the issue, focusing on the via = "drive", lazy = TRUE path and ee_utils_future_value(). Reproduce the Terraclimate extraction and compare it with the getInfo path; done means identifying a reproducible cause or documenting that the Drive extraction completes reliably.

Written by the indexing model from the issue text.

Description

  • rgee version:
  • R version:
  • Operating System:
At submit an issue, please attached the following information of your rgee session:
  • You have the Python API installed (from terminal):
earthengine -h
  • You can find the credentials file on your system:
library(rgee)
ee_path <- path.expand("~/.config/earthengine/credentials")
file.exists(ee_path)
  • You can run a simple EE command from R:
library(rgee)

# Initialize the Earth Engine module.
ee_Initialize()

# Print metadata for a DEM dataset.
print(ee$Image('USGS/SRTMGL1_003')$getInfo())

Attach your Python (reticulate) configuration:

> library(reticulate)
Warning message:
package ‘reticulate’ was built under R version 4.2.3 
> py_config()
python:         C:/Users/00057/AppData/Local/r-miniconda/envs/rgee/python.exe
libpython:      C:/Users/00057/AppData/Local/r-miniconda/envs/rgee/python38.dll
pythonhome:     C:/Users/00057/AppData/Local/r-miniconda/envs/rgee
version:        3.8.17 | packaged by conda-forge | (default, Jun 16 2023, 07:01:59) [MSC v.1929 64 bit (AMD64)]
Architecture:   64bit
numpy:          C:/Users/00057/AppData/Local/r-miniconda/envs/rgee/Lib/site-packages/numpy
numpy_version:  1.24.4
ee:             C:\Users\00057\AppData\Local\R-MINI~1\envs\rgee\lib\site-packages\ee\__init__.p

NOTE: Python version was forced by RETICULATE_PYTHON
Description

EDIT: This problem seems to have disappeared on it's own rerunning the code a week later. Could this be a server-side problem, e.g. rate limiting or GEE being too heavily used at the time?

If I try and extract data with the google drive option, the extraction lags forever. However, if I try the getInfo option, it is almost instantaneous. This happens with every dataset I've tried. I've attached the example from the docs found at https://r-spatial.github.io/rgee/reference/ee_extract.html, below.

What I Did

The example found at https://r-spatial.github.io/rgee/reference/ee_extract.html

library(rgee)
library(sf)

ee_Initialize(gcs = TRUE, drive = TRUE)

# Define a Image or ImageCollection: Terraclimate
terraclimate <- ee$ImageCollection("IDAHO_EPSCOR/TERRACLIMATE") %>%
 ee$ImageCollection$filterDate("2001-01-01", "2002-01-01") %>%
ee$ImageCollection$map(
   function(x) {
     date <- ee$Date(x$get("system:time_start"))$format('YYYY_MM_dd')
     name <- ee$String$cat("Terraclimate_pp_", date)
     x$select("pr")$rename(name)
   }
 )

# Define a geometry
nc <- st_read(
 dsn = system.file("shape/nc.shp", package = "sf"),
 stringsAsFactors = FALSE,
 quiet = TRUE
)


#Extract values - getInfo
# this works fine and very fast
ee_nc_rain <- ee_extract(
 x = terraclimate,
 y = nc["NAME"],
 scale = 250,
 fun = ee$Reducer$mean(),
 sf = TRUE
)

# Extract values - drive (lazy = TRUE)
# this starts and never finishes
ee_nc_rain <- ee_extract(
 x = terraclimate,
 y = nc["NAME"],
 scale = 250,
 fun = ee$Reducer$mean(),
 via = "drive",
 lazy = TRUE,
 sf = TRUE
)
ee_nc_rain <- ee_nc_rain %>% ee_utils_future_value()
Dominant language
R
Stars
777
Forks
159
PR merge metrics
No merged PRs in 30d

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from r-spatial/rgee

All issues in r-spatial/rgee

Similar issues

More R issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.