Sentinel-5 P data usage

Offen
#303 5 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Dieses Issue hat noch niemand übernommen.

Bewertung

Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Anfängerfreundlichkeit
35/100
Issue-Typ
Bug
Klarheit
Muss geklärt werden
Aktivitätsstatus
Veraltet
Tech-Stack
jupyter-notebook, python
Bereich
data

Rechercherichtung

Beginne mit dem Notebook-Code, der die Planetary Computer STAC API durchsucht und die ch4-Assets an stackstac.stack übergibt; untersuche die generierten Sentinel-5P-Blob-URLs und die gemeldeten 404-Antworten. Bestätige, ob die STAC-Asset-Links gültig sind, und ermittle das erwartete Verhalten, wenn der Stack persistiert und zugeschnitten wird, ohne dass beim Öffnen der Assets Fehler auftreten.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Beschreibung

Hi! I've been trying to use sentinel-5 data to map CH4 emissions over my area of interest.

I replicated pretty much the code that i previously had for sentinel 1 and sentinel 2 processing:

sentinel_search_url = "https://planetarycomputer.microsoft.com/api/stac/v1"
sentinel_stac_client = pystac_client.Client.open(sentinel_search_url)

items = sentinel_stac_client.search(
    bbox=bounds,
    collections=["sentinel-5p-l2-netcdf"],
    datetime=f"{start_date}/{end_date}",
    query={"s5p:processing_mode": {"eq": "OFFL"}, "s5p:product_name": {"eq": "ch4"}},
).item_collection()

print(f"Total number of items found: {len(items)}")

scene_dates = sorted([item.datetime for item in items])

selected_scenes = [scene_dates[0]]
for i in range(1, len(scene_dates)):
    if (scene_dates[i] - selected_scenes[-1]).days >= min_time_step:
        selected_scenes.append(scene_dates[i])

items = [
    item for item in items if item.datetime in selected_scenes
]

print(f"Number of items after minimum time step filtering: {len(items)}")

print("\nSelected scenes after filtering:")
for item in items:
    print(f"Scene date: {item.datetime}")


(i had to provide a proj:epsg field as this data doesn't have it)

for item in items:
    # Access the asset metadata
    asset = item.assets['ch4']
    
    # Add proj:epsg metadata if it doesn't exist
    if 'proj:epsg' not in asset.extra_fields:
        asset.extra_fields['proj:epsg'] = 4326

sentinel_stack = stackstac.stack(
    items,
    assets=["ch4"],
    gdal_env=stackstac.DEFAULT_GDAL_ENV.updated({
        'GDAL_HTTP_MAX_RETRY': 3,
        'GDAL_HTTP_RETRY_DELAY': 5,
        'AWS_NO_SIGN_REQUEST': 'YES',
    }),
    epsg=4326,
    resolution=0.0629, 
    chunksize=(1, 1, 1000, 1000),
).to_dataset(dim='band')

# This length number represents the number  of assets (bands) that are to be extracted
# len(sentinel_stack)

sentinel_stack

# Remove attributes that are not time, y or x 
sentinel_stack = sentinel_stack.drop_vars([c for c in sentinel_stack.coords if not (c in ['time', 'y', 'x',])])
sentinel_stack

sentinel_stack_float32 = sentinel_stack.astype('float32')
sentinel_stack_float32

crs = "EPSG:4326"
crs_number = crs[5:]
sentinel_stack_float32 = sentinel_stack_float32.rio.write_crs(fr"{crs}", inplace=True)
sentinel_stack_float32

sentinel_stack_clipped = sentinel_stack_float32.rio.clip(aoi.geometry, aoi.crs, all_touched=True, drop=True)
with ProgressBar():
    sentinel_stack_clipped = sentinel_stack_clipped.persist()

Here, i get:

############ ] | 32% Completed | 1.75 sms
c:\Users\fuji_\anaconda3\envs\geo\Lib\site-packages\stackstac\rio_reader.py:329: UserWarning: Error opening 'https://sentinel5euwest.blob.core.windows.net/sentinel-5p/TROPOMI/L2__CH4___/2024/01/01/S5P_OFFL_L2__CH4____20240101T092629_20240101T110759_32220_03_020600_20240103T013302/S5P_OFFL_L2__CH4____20240101T092629_20240101T110759_32220_03_020600_20240103T013302.nc': RasterioIOError('HTTP response code: 404')
warnings.warn(msg)
c:\Users\fuji_\anaconda3\envs\geo\Lib\site-packages\stackstac\rio_reader.py:329: UserWarning: Error opening 'https://sentinel5euwest.blob.core.windows.net/sentinel-5p/TROPOMI/L2__CH4___/2024/01/06/S5P_OFFL_L2__CH4____20240106T093259_20240106T111429_32291_03_020600_20240108T014844/S5P_OFFL_L2__CH4____20240106T093259_20240106T111429_32291_03_020600_20240108T014844.nc': RasterioIOError('HTTP response code: 404')
warnings.warn(msg)
c:\Users\fuji_\anaconda3\envs\geo\Lib\site-packages\stackstac\rio_reader.py:329: UserWarning: Error opening 'https://sentinel5euwest.blob.core.windows.net/sentinel-5p/TROPOMI/L2__CH4___/2024/03/17/S5P_OFFL_L2__CH4____20240317T104914_20240317T123044_33299_03_020600_20240319T160340/S5P_OFFL_L2__CH4____20240317T104914_20240317T123044_33299_03_020600_20240319T160340.nc': RasterioIOError('HTTP response code: 404')
warnings.warn(msg)
c:\Users\fuji_\anaconda3\envs\geo\Lib\site-packages\stackstac\rio_reader.py:329: UserWarning: Error opening 'https://sentinel5euwest.blob.core.windows.net/sentinel-5p/TROPOMI/L2__CH4___/2024/04/19/S5P_OFFL_L2__CH4____20240419T103414_20240419T121544_33767_03_020600_20240421T231632/S5P_OFFL_L2__CH4____20240419T103414_20240419T121544_33767_03_020600_20240421T231632.nc': RasterioIOError('HTTP response code: 404')
warnings.warn(msg)
c:\Users\fuji_\anaconda3\envs\geo\Lib\site-packages\stackstac\rio_reader.py:329: UserWarning: Error opening 'https://sentinel5euwest.blob.core.windows.net/sentinel-5p/TROPOMI/L2__CH4___/2024/03/27/S5P_OFFL_L2__CH4____20240327T110315_20240327T124445_33441_03_020600_20240329T115341/S5P_OFFL_L2__CH4____20240327T110315_20240327T124445_33441_03_020600_20240329T115341.nc': RasterioIOError('HTTP response code: 404')
warnings.warn(msg)
c:\Users\fuji_\anaconda3\envs\geo\Lib\site-packages\stackstac\rio_reader.py:329: UserWarning: Error opening 'https://sentinel5euwest.blob.core.windows.net/sentinel-5p/TROPOMI/L2__CH4___/2024/03/22/S5P_OFFL_L2__CH4____20240322T105615_20240322T123745_33370_03_020600_20240324T111434/S5P_OFFL_L2__CH4____20240322T105615_20240322T123745_33370_03_020600_20240324T111434.nc': RasterioIOError('HTTP response code: 404')
warnings.warn(msg)
c:\Users\fuji_\anaconda3\envs\geo\Lib\site-packages\stackstac\rio_reader.py:329: UserWarning: Error opening 'https://sentinel5euwest.blob.core.windows.net/sentinel-5p/TROPOMI/L2__CH4___/2024/02/26/S5P_OFFL_L2__CH4____20240226T102115_20240226T120245_33015_03_020600_20240228T022449/S5P_OFFL_L2__CH4____20240226T102115_20240226T120245_33015_03_020600_20240228T022449.nc': RasterioIOError('HTTP response code: 404')
warnings.warn(msg)
c:\Users\fuji_\anaconda3\envs\geo\Lib\site-packages\stackstac\rio_reader.py:329: UserWarning: Error opening 'https://sentinel5euwest.blob.core.windows.net/sentinel-5p/TROPOMI/L2__CH4___/2024/04/01/S5P_OFFL_L2__CH4____20240401T111013_20240401T125144_33512_03_020600_20240403T032212/S5P_OFFL_L2__CH4____20240401T111013_20240401T125144_33512_03_020600_20240403T032212.nc': RasterioIOError('HTTP response code: 404')
warnings.warn(msg)
c:\Users\fuji_\anaconda3\envs\geo\Lib\site-packages\stackstac\rio_reader.py:329: UserWarning: Error opening 'https://sentinel5euwest.blob.core.windows.net/sentinel-5p/TROPOMI/L2__CH4___/2024/02/21/S5P_OFFL_L2__CH4____20240221T101417_20240221T115547_32944_03_020600_20240223T023451/S5P_OFFL_L2__CH4____20240221T101417_20240221T115547_32944_03_020600_20240223T023451.nc': RasterioIOError('HTTP response code: 404')
warnings.warn(msg)
c:\Users\fuji_\anaconda3\envs\geo\Lib\site-packages\stackstac\rio_reader.py:329: UserWarning: Error opening 'https://sentinel5euwest.blob.core.windows.net/sentinel-5p/TROPOMI/L2__CH4___/2024/01/11/S5P_OFFL_L2__CH4____20240111T093933_20240111T112103_32362_03_020600_20240113T020813/S5P_OFFL_L2__CH4____20240111T093933_20240111T112103_32362_03_020600_20240113T020813.nc': RasterioIOError('HTTP response code: 404')
warnings.warn(msg)
c:\Users\fuji_\anaconda3\envs\geo\Lib\site-packages\stackstac\rio_reader.py:329: UserWarning: Error opening 'https://sentinel5euwest.blob.core.windows.net/sentinel-5p/TROPOMI/L2__CH4___/2024/03/07/S5P_OFFL_L2__CH4____20240307T103513_20240307T121644_33157_03_020600_20240309T030532/S5P_OFFL_L2__CH4____20240307T103513_20240307T121644_33157_03_020600_20240309T030532.nc': RasterioIOError('HTTP response code: 404')
warnings.warn(msg)
c:\Users\fuji_\anaconda3\envs\geo\Lib\site-packages\stackstac\rio_reader.py:329: UserWarning: Error opening 'https://sentinel5euwest.blob.core.windows.net/sentinel-5p/TROPOMI/L2__CH4___/2024/03/02/S5P_OFFL_L2__CH4____20240302T102814_20240302T120944_33086_03_020600_20240304T025010/S5P_OFFL_L2__CH4____20240302T102814_20240302T120944_33086_03_020600_20240304T025010.nc': RasterioIOError('HTTP response code: 404')
warnings.warn(msg)
c:\Users\fuji_\anaconda3\envs\geo\Lib\site-packages\stackstac\rio_reader.py:329: UserWarning: Error opening 'https://sentinel5euwest.blob.core.windows.net/sentinel-5p/TROPOMI/L2__CH4___/2024/04/08/S5P_OFFL_L2__CH4____20240408T103921_20240408T122051_33611_03_020600_20240410T030005/S5P_OFFL_L2__CH4____20240408T103921_20240408T122051_33611_03_020600_20240410T030005.nc': RasterioIOError('HTTP response code: 404')
...
c:\Users\fuji_\anaconda3\envs\geo\Lib\site-packages\stackstac\rio_reader.py:329: UserWarning: Error opening 'https://sentinel5euwest.blob.core.windows.net/sentinel-5p/TROPOMI/L2__CH4___/2024/01/21/S5P_OFFL_L2__CH4____20240121T095247_20240121T113418_32504_03_020600_20240123T022011/S5P_OFFL_L2__CH4____20240121T095247_20240121T113418_32504_03_020600_20240123T022011.nc': RasterioIOError('HTTP response code: 404')
warnings.warn(msg)
c:\Users\fuji_\anaconda3\envs\geo\Lib\site-packages\stackstac\rio_reader.py:329: UserWarning: Error opening 'https://sentinel5euwest.blob.core.windows.net/sentinel-5p/TROPOMI/L2__CH4___/2024/07/24/S5P_OFFL_L2__CH4____20240724T103609_20240724T121739_35129_03_020600_20240726T072607/S5P_OFFL_L2__CH4____20240724T103609_20240724T121739_35129_03_020600_20240726T072607.nc': RasterioIOError('HTTP response code: 404')
warnings.warn(msg)
Output is truncated. View as a scrollable element or open in a text editor. Adjust cell output settings...
[################################## ] | 85% Completed | 1.96 s
c:\Users\fuji_\anaconda3\envs\geo\Lib\site-packages\stackstac\rio_reader.py:329: UserWarning: Error opening 'https://sentinel5euwest.blob.core.windows.net/sentinel-5p/TROPOMI/L2__CH4___/2024/07/29/S5P_OFFL_L2__CH4____20240729T104210_20240729T122340_35200_03_020600_20240804T092141/S5P_OFFL_L2__CH4____20240729T104210_20240729T122340_35200_03_020600_20240804T092141.nc': RasterioIOError('HTTP response code: 404')
warnings.warn(msg)
c:\Users\fuji_\anaconda3\envs\geo\Lib\site-packages\stackstac\rio_reader.py:329: UserWarning: Error opening 'https://sentinel5euwest.blob.core.windows.net/sentinel-5p/TROPOMI/L2__CH4___/2024/08/03/S5P_OFFL_L2__CH4____20240803T104809_20240803T122938_35271_03_020600_20240809T130432/S5P_OFFL_L2__CH4____20240803T104809_20240803T122938_35271_03_020600_20240809T130432.nc': RasterioIOError('HTTP response code: 404')
warnings.warn(msg)
c:\Users\fuji_\anaconda3\envs\geo\Lib\site-packages\stackstac\rio_reader.py:329: UserWarning: Error opening 'https://sentinel5euwest.blob.core.windows.net/sentinel-5p/TROPOMI/L2__CH4___/2024/08/08/S5P_OFFL_L2__CH4____20240808T105404_20240808T123534_35342_03_020600_20240812T214910/S5P_OFFL_L2__CH4____20240808T105404_20240808T123534_35342_03_020600_20240812T214910.nc': RasterioIOError('HTTP response code: 404')
warnings.warn(msg)
c:\Users\fuji_\anaconda3\envs\geo\Lib\site-packages\stackstac\rio_reader.py:329: UserWarning: Error opening 'https://sentinel5euwest.blob.core.windows.net/sentinel-5p/TROPOMI/L2__CH4___/2024/08/18/S5P_OFFL_L2__CH4____20240818T110547_20240818T124717_35484_03_020600_20240820T031012/S5P_OFFL_L2__CH4____20240818T110547_20240818T124717_35484_03_020600_20240820T031012.nc': RasterioIOError('HTTP response code: 404')
warnings.warn(msg)
c:\Users\fuji_\anaconda3\envs\geo\Lib\site-packages\stackstac\rio_reader.py:329: UserWarning: Error opening 'https://sentinel5euwest.blob.core.windows.net/sentinel-5p/TROPOMI/L2__CH4___/2024/08/23/S5P_OFFL_L2__CH4____20240823T111134_20240823T125304_35555_03_020600_20240825T033549/S5P_OFFL_L2__CH4____20240823T111134_20240823T125304_35555_03_020600_20240825T033549.nc': RasterioIOError('HTTP response code: 404')
warnings.warn(msg)
c:\Users\fuji_\anaconda3\envs\geo\Lib\site-packages\stackstac\rio_reader.py:329: UserWarning: Error opening 'https://sentinel5euwest.blob.core.windows.net/sentinel-5p/TROPOMI/L2__CH4___/2024/08/28/S5P_OFFL_L2__CH4____20240828T111719_20240828T125848_35626_03_020600_20240830T032246/S5P_OFFL_L2__CH4____20240828T111719_20240828T125848_35626_03_020600_20240830T032246.nc': RasterioIOError('HTTP response code: 404')
warnings.warn(msg)
c:\Users\fuji_\anaconda3\envs\geo\Lib\site-packages\stackstac\rio_reader.py:329: UserWarning: Error opening 'https://sentinel5euwest.blob.core.windows.net/sentinel-5p/TROPOMI/L2__CH4___/2024/09/02/S5P_OFFL_L2__CH4____20240902T112301_20240902T130431_35697_03_020600_20240904T034141/S5P_OFFL_L2__CH4____20240902T112301_20240902T130431_35697_03_020600_20240904T034141.nc': RasterioIOError('HTTP response code: 404')
warnings.warn(msg)
c:\Users\fuji_\anaconda3\envs\geo\Lib\site-packages\stackstac\rio_reader.py:329: UserWarning: Error opening 'https://sentinel5euwest.blob.core.windows.net/sentinel-5p/TROPOMI/L2__CH4___/2024/09/07/S5P_OFFL_L2__CH4____20240907T112851_20240907T131021_35768_03_020600_20240909T040338/S5P_OFFL_L2__CH4____20240907T112851_20240907T131021_35768_03_020600_20240909T040338.nc': RasterioIOError('HTTP response code: 404')
warnings.warn(msg)
c:\Users\fuji_\anaconda3\envs\geo\Lib\site-packages\stackstac\rio_reader.py:329: UserWarning: Error opening 'https://sentinel5euwest.blob.core.windows.net/sentinel-5p/TROPOMI/L2__CH4___/2024/09/17/S5P_OFFL_L2__CH4____20240917T114040_20240917T132210_35910_03_020701_20240919T063955/S5P_OFFL_L2__CH4____20240917T114040_20240917T132210_35910_03_020701_20240919T063955.nc': RasterioIOError('HTTP response code: 404')
warnings.warn(msg)
c:\Users\fuji_\anaconda3\envs\geo\Lib\site-packages\stackstac\rio_reader.py:329: UserWarning: Error opening 'https://sentinel5euwest.blob.core.windows.net/sentinel-5p/TROPOMI/L2__CH4___/2024/08/13/S5P_OFFL_L2__CH4____20240813T105958_20240813T124127_35413_03_020600_20240815T215037/S5P_OFFL_L2__CH4____20240813T105958_20240813T124127_35413_03_020600_20240815T215037.nc': RasterioIOError('HTTP response code: 404')
warnings.warn(msg)
c:\Users\fuji_\anaconda3\envs\geo\Lib\site-packages\stackstac\rio_reader.py:329: UserWarning: Error opening 'https://sentinel5euwest.blob.core.windows.net/sentinel-5p/TROPOMI/L2__CH4___/2024/09/22/S5P_OFFL_L2__CH4____20240922T114632_20240922T132801_35981_03_020701_20240924T071754/S5P_OFFL_L2__CH4____20240922T114632_20240922T132801_35981_03_020701_20240924T071754.nc': RasterioIOError('HTTP response code: 404')
warnings.warn(msg)
c:\Users\fuji_\anaconda3\envs\geo\Lib\site-packages\stackstac\rio_reader.py:329: UserWarning: Error opening 'https://sentinel5euwest.blob.core.windows.net/sentinel-5p/TROPOMI/L2__CH4___/2024/09/12/S5P_OFFL_L2__CH4____20240912T113447_20240912T131617_35839_03_020701_20240914T201419/S5P_OFFL_L2__CH4____20240912T113447_20240912T131617_35839_03_020701_20240914T201419.nc': RasterioIOError('HTTP response code: 404')
warnings.warn(msg)
c:\Users\fuji_\anaconda3\envs\geo\Lib\site-packages\stackstac\rio_reader.py:329: UserWarning: Error opening 'https://sentinel5euwest.blob.core.windows.net/sentinel-5p/TROPOMI/L2__CH4___/2024/01/26/S5P_OFFL_L2__CH4____20240126T114058_20240126T132228_32576_03_020600_20240128T051841/S5P_OFFL_L2__CH4____20240126T114058_20240126T132228_32576_03_020600_20240128T051841.nc': RasterioIOError('HTTP response code: 404')
...
c:\Users\fuji_\anaconda3\envs\geo\Lib\site-packages\stackstac\rio_reader.py:329: UserWarning: Error opening 'https://sentinel5euwest.blob.core.windows.net/sentinel-5p/TROPOMI/L2__CH4___/2024/11/12/S5P_OFFL_L2__CH4____20241112T104321_20241112T122450_36704_03_020701_20241115T183324/S5P_OFFL_L2__CH4____20241112T104321_20241112T122450_36704_03_020701_20241115T183324.nc': RasterioIOError('HTTP response code: 404')
warnings.warn(msg)
c:\Users\fuji_\anaconda3\envs\geo\Lib\site-packages\stackstac\rio_reader.py:329: UserWarning: Error opening 'https://sentinel5euwest.blob.core.windows.net/sentinel-5p/TROPOMI/L2__CH4___/2024/12/17/S5P_OFFL_L2__CH4____20241217T094534_20241217T112704_37200_03_020800_20241219T020255/S5P_OFFL_L2__CH4____20241217T094534_20241217T112704_37200_03_020800_20241219T020255.nc': RasterioIOError('HTTP response code: 404')
warnings.warn(msg)
Output is truncated. View as a scrollable element or open in a text editor. Adjust cell output settings...
[########################################] | 100% Completed | 2.06 s
c:\Users\fuji_\anaconda3\envs\geo\Lib\site-packages\stackstac\rio_reader.py:329: UserWarning: Error opening 'https://sentinel5euwest.blob.core.windows.net/sentinel-5p/TROPOMI/L2__CH4___/2024/02/05/S5P_OFFL_L2__CH4____20240205T115424_20240205T133554_32718_03_020600_20240207T042306/S5P_OFFL_L2__CH4____20240205T115424_20240205T133554_32718_03_020600_20240207T042306.nc': RasterioIOError('HTTP response code: 404')
warnings.warn(msg)
c:\Users\fuji_\anaconda3\envs\geo\Lib\site-packages\stackstac\rio_reader.py:329: UserWarning: Error opening 'https://sentinel5euwest.blob.core.windows.net/sentinel-5p/TROPOMI/L2__CH4___/2024/02/10/S5P_OFFL_L2__CH4____20240210T120110_20240210T134240_32789_03_020600_20240212T042117/S5P_OFFL_L2__CH4____20240210T120110_20240210T134240_32789_03_020600_20240212T042117.nc': RasterioIOError('HTTP response code: 404')
warnings.warn(msg)
c:\Users\fuji_\anaconda3\envs\geo\Lib\site-packages\stackstac\rio_reader.py:329: UserWarning: Error opening 'https://sentinel5euwest.blob.core.windows.net/sentinel-5p/TROPOMI/L2__CH4___/2024/02/16/S5P_OFFL_L2__CH4____20240216T100729_20240216T114900_32873_03_020600_20240218T023150/S5P_OFFL_L2__CH4____20240216T100729_20240216T114900_32873_03_020600_20240218T023150.nc': RasterioIOError('HTTP response code: 404')
warnings.warn(msg)

And when i try to save the datarray by computing the mean over this clipped area, i get a tiff file with no results

Vorherrschende Sprache
Jupyter Notebook
Sterne
455
Forks
225
PR-Merge-Kennzahlen
Keine gemergten PRs in 30 T.

Beitragsleitfaden

Beitragsleitfaden öffnen

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus microsoft/PlanetaryComputerExamples

Alle Issues in microsoft/PlanetaryComputerExamples

Ähnliche Issues

Weitere Issues zu Data Engineering

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.