nasa / nasa/Common-Metadata-Repository
Unable to get "next page of results" when searching STAC features
Nobody has claimed this yet.
- Dominant language
- Clojure
- Stars
- 397
- Forks
- 108
- Avg merge
- 3d 14m
- Merged PRs (30d)
- 13
Description
Hello,
I'm trying to do a search on https://cmr.earthdata.nasa.gov/stac/LAADS/search (using Python). My code was working until a few weeks ago, but now I'm not able to retrieve all results when the number of items returned exceeds limit. Here is an example:
import requests
url = 'https://cmr.earthdata.nasa.gov/stac/LAADS/search'
params = {
'limit': 250,
'bbox': [
36.42191632061822,
32.24152077745001,
36.56950917300816,
32.37621048001086
],
'datetime': '2023-03-01T00:00:00Z/2023-09-30T23:59:59Z',
'collections': ['VNP03IMG.v2']
}
resp = requests.post(url, json = params)
resp.raise_for_status()
output_01 = resp.json()
next_page = [x for x in output_01["links"] if x["rel"] == "next"][0]
print(len(output_01["features"]))
print(next_page["title"])
print(next_page["href"])
>>> 250
>>> Next page of results
>>> https://cmr.earthdata.nasa.gov/stac/LAADS/search?limit=250&bbox=36.42191632061822%2C32.24152077745001%2C36.56950917300816%2C32.37621048001086&datetime=2023-03-01T00%3A00%3A00Z%2F2023-09-30T23%3A59%3A59Z&collections=VNP03IMG.v2&cursor=eyJqc29uIjoiW1wibGFhZHNcIiwxNjg1NzUxODQwMDAwLDI3MDI2NjQ3NDFdIiwidW1tIjoiW1wibGFhZHNcIiwxNjg1NzUxODQwMDAwLDI3MDI2NjQ3NDFdIn0%3D
So far all is good, I've got 250 features, but now I need to request the next page. Both methods below (using either GET or POST) give me a 500 response:
resp = requests.get(next_page["href"])
resp.raise_for_status()
resp = requests.post(next_page["href"], json = params)
resp.raise_for_status()
Any advice on how to continue?
Thanks!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the POST search against the CMR STAC endpoint shown in the Python example, then follow the returned next link with both GET and POST. Compare the cursor and request parameters in each response; done means the next-page request returns the remaining features instead of a 500 error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, search
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100