Sentinel-5 P data usage

Hi! I've been trying to use sentinel-5 data to map CH4 emissions over my area of interest.

I replicated pretty much the code that i previously had for sentinel 1 and sentinel 2 processing:

sentinel_search_url = "https://planetarycomputer.microsoft.com/api/stac/v1"
sentinel_stac_client = pystac_client.Client.open(sentinel_search_url)

items = sentinel_stac_client.search(
    bbox=bounds,
    collections=["sentinel-5p-l2-netcdf"],
    datetime=f"{start_date}/{end_date}",
    query={"s5p:processing_mode": {"eq": "OFFL"}, "s5p:product_name": {"eq": "ch4"}},
).item_collection()

print(f"Total number of items found: {len(items)}")

scene_dates = sorted([item.datetime for item in items])

selected_scenes = [scene_dates[0]]
for i in range(1, len(scene_dates)):
    if (scene_dates[i] - selected_scenes[-1]).days >= min_time_step:
        selected_scenes.append(scene_dates[i])

items = [
    item for item in items if item.datetime in selected_scenes
]

print(f"Number of items after minimum time step filtering: {len(items)}")

print("\nSelected scenes after filtering:")
for item in items:
    print(f"Scene date: {item.datetime}")


(i had to provide a proj:epsg field as this data doesn't have it)

for item in items:
    # Access the asset metadata
    asset = item.assets['ch4']
    
    # Add proj:epsg metadata if it doesn't exist
    if 'proj:epsg' not in asset.extra_fields:
        asset.extra_fields['proj:epsg'] = 4326

sentinel_stack = stackstac.stack(
    items,
    assets=["ch4"],
    gdal_env=stackstac.DEFAULT_GDAL_ENV.updated({
        'GDAL_HTTP_MAX_RETRY': 3,
        'GDAL_HTTP_RETRY_DELAY': 5,
        'AWS_NO_SIGN_REQUEST': 'YES',
    }),
    epsg=4326,
    resolution=0.0629, 
    chunksize=(1, 1, 1000, 1000),
).to_dataset(dim='band')

# This length number represents the number  of assets (bands) that are to be extracted
# len(sentinel_stack)

sentinel_stack

# Remove attributes that are not time, y or x 
sentinel_stack = sentinel_stack.drop_vars([c for c in sentinel_stack.coords if not (c in ['time', 'y', 'x',])])
sentinel_stack

sentinel_stack_float32 = sentinel_stack.astype('float32')
sentinel_stack_float32

crs = "EPSG:4326"
crs_number = crs[5:]
sentinel_stack_float32 = sentinel_stack_float32.rio.write_crs(fr"{crs}", inplace=True)
sentinel_stack_float32

sentinel_stack_clipped = sentinel_stack_float32.rio.clip(aoi.geometry, aoi.crs, all_touched=True, drop=True)
with ProgressBar():
    sentinel_stack_clipped = sentinel_stack_clipped.persist()

Here, i get:

############ ] | 32% Completed | 1.75 sms
c:\Users\fuji_\anaconda3\envs\geo\Lib\site-packages\stackstac\rio_reader.py:329: UserWarning: Error opening 'https://sentinel5euwest.blob.core.windows.net/sentinel-5p/TROPOMI/L2__CH4___/2024/01/01/S5P_OFFL_L2__CH4____20240101T092629_20240101T110759_32220_03_020600_20240103T013302/S5P_OFFL_L2__CH4____20240101T092629_20240101T110759_32220_03_020600_20240103T013302.nc': RasterioIOError('HTTP response code: 404')
warnings.warn(msg)
c:\Users\fuji_\anaconda3\envs\geo\Lib\site-packages\stackstac\rio_reader.py:329: UserWarning: Error opening 'https://sentinel5euwest.blob.core.windows.net/sentinel-5p/TROPOMI/L2__CH4___/2024/01/06/S5P_OFFL_L2__CH4____20240106T093259_20240106T111429_32291_03_020600_20240108T014844/S5P_OFFL_L2__CH4____20240106T093259_20240106T111429_32291_03_020600_20240108T014844.nc': RasterioIOError('HTTP response code: 404')
warnings.warn(msg)
c:\Users\fuji_\anaconda3\envs\geo\Lib\site-packages\stackstac\rio_reader.py:329: UserWarning: Error opening 'https://sentinel5euwest.blob.core.windows.net/sentinel-5p/TROPOMI/L2__CH4___/2024/03/17/S5P_OFFL_L2__CH4____20240317T104914_20240317T123044_33299_03_020600_20240319T160340/S5P_OFFL_L2__CH4____20240317T104914_20240317T123044_33299_03_020600_20240319T160340.nc': RasterioIOError('HTTP response code: 404')
warnings.warn(msg)
c:\Users\fuji_\anaconda3\envs\geo\Lib\site-packages\stackstac\rio_reader.py:329: UserWarning: Error opening 'https://sentinel5euwest.blob.core.windows.net/sentinel-5p/TROPOMI/L2__CH4___/2024/04/19/S5P_OFFL_L2__CH4____20240419T103414_20240419T121544_33767_03_020600_20240421T231632/S5P_OFFL_L2__CH4____20240419T103414_20240419T121544_33767_03_020600_20240421T231632.nc': RasterioIOError('HTTP response code: 404')
warnings.warn(msg)
c:\Users\fuji_\anaconda3\envs\geo\Lib\site-packages\stackstac\rio_reader.py:329: UserWarning: Error opening 'https://sentinel5euwest.blob.core.windows.net/sentinel-5p/TROPOMI/L2__CH4___/2024/03/27/S5P_OFFL_L2__CH4____20240327T110315_20240327T124445_33441_03_020600_20240329T115341/S5P_OFFL_L2__CH4____20240327T110315_20240327T124445_33441_03_020600_20240329T115341.nc': RasterioIOError('HTTP response code: 404')
warnings.warn(msg)
c:\Users\fuji_\anaconda3\envs\geo\Lib\site-packages\stackstac\rio_reader.py:329: UserWarning: Error opening 'https://sentinel5euwest.blob.core.windows.net/sentinel-5p/TROPOMI/L2__CH4___/2024/03/22/S5P_OFFL_L2__CH4____20240322T105615_20240322T123745_33370_03_020600_20240324T111434/S5P_OFFL_L2__CH4____20240322T105615_20240322T123745_33370_03_020600_20240324T111434.nc': RasterioIOError('HTTP response code: 404')
warnings.warn(msg)
c:\Users\fuji_\anaconda3\envs\geo\Lib\site-packages\stackstac\rio_reader.py:329: UserWarning: Error opening 'https://sentinel5euwest.blob.core.windows.net/sentinel-5p/TROPOMI/L2__CH4___/2024/02/26/S5P_OFFL_L2__CH4____20240226T102115_20240226T120245_33015_03_020600_20240228T022449/S5P_OFFL_L2__CH4____20240226T102115_20240226T120245_33015_03_020600_20240228T022449.nc': RasterioIOError('HTTP response code: 404')
warnings.warn(msg)
c:\Users\fuji_\anaconda3\envs\geo\Lib\site-packages\stackstac\rio_reader.py:329: UserWarning: Error opening 'https://sentinel5euwest.blob.core.windows.net/sentinel-5p/TROPOMI/L2__CH4___/2024/04/01/S5P_OFFL_L2__CH4____20240401T111013_20240401T125144_33512_03_020600_20240403T032212/S5P_OFFL_L2__CH4____20240401T111013_20240401T125144_33512_03_020600_20240403T032212.nc': RasterioIOError('HTTP response code: 404')
warnings.warn(msg)
c:\Users\fuji_\anaconda3\envs\geo\Lib\site-packages\stackstac\rio_reader.py:329: UserWarning: Error opening 'https://sentinel5euwest.blob.core.windows.net/sentinel-5p/TROPOMI/L2__CH4___/2024/02/21/S5P_OFFL_L2__CH4____20240221T101417_20240221T115547_32944_03_020600_20240223T023451/S5P_OFFL_L2__CH4____20240221T101417_20240221T115547_32944_03_020600_20240223T023451.nc': RasterioIOError('HTTP response code: 404')
warnings.warn(msg)
c:\Users\fuji_\anaconda3\envs\geo\Lib\site-packages\stackstac\rio_reader.py:329: UserWarning: Error opening 'https://sentinel5euwest.blob.core.windows.net/sentinel-5p/TROPOMI/L2__CH4___/2024/01/11/S5P_OFFL_L2__CH4____20240111T093933_20240111T112103_32362_03_020600_20240113T020813/S5P_OFFL_L2__CH4____20240111T093933_20240111T112103_32362_03_020600_20240113T020813.nc': RasterioIOError('HTTP response code: 404')
warnings.warn(msg)
c:\Users\fuji_\anaconda3\envs\geo\Lib\site-packages\stackstac\rio_reader.py:329: UserWarning: Error opening 'https://sentinel5euwest.blob.core.windows.net/sentinel-5p/TROPOMI/L2__CH4___/2024/03/07/S5P_OFFL_L2__CH4____20240307T103513_20240307T121644_33157_03_020600_20240309T030532/S5P_OFFL_L2__CH4____20240307T103513_20240307T121644_33157_03_020600_20240309T030532.nc': RasterioIOError('HTTP response code: 404')
warnings.warn(msg)
c:\Users\fuji_\anaconda3\envs\geo\Lib\site-packages\stackstac\rio_reader.py:329: UserWarning: Error opening 'https://sentinel5euwest.blob.core.windows.net/sentinel-5p/TROPOMI/L2__CH4___/2024/03/02/S5P_OFFL_L2__CH4____20240302T102814_20240302T120944_33086_03_020600_20240304T025010/S5P_OFFL_L2__CH4____20240302T102814_20240302T120944_33086_03_020600_20240304T025010.nc': RasterioIOError('HTTP response code: 404')
warnings.warn(msg)
c:\Users\fuji_\anaconda3\envs\geo\Lib\site-packages\stackstac\rio_reader.py:329: UserWarning: Error opening 'https://sentinel5euwest.blob.core.windows.net/sentinel-5p/TROPOMI/L2__CH4___/2024/04/08/S5P_OFFL_L2__CH4____20240408T103921_20240408T122051_33611_03_020600_20240410T030005/S5P_OFFL_L2__CH4____20240408T103921_20240408T122051_33611_03_020600_20240410T030005.nc': RasterioIOError('HTTP response code: 404')
...
c:\Users\fuji_\anaconda3\envs\geo\Lib\site-packages\stackstac\rio_reader.py:329: UserWarning: Error opening 'https://sentinel5euwest.blob.core.windows.net/sentinel-5p/TROPOMI/L2__CH4___/2024/01/21/S5P_OFFL_L2__CH4____20240121T095247_20240121T113418_32504_03_020600_20240123T022011/S5P_OFFL_L2__CH4____20240121T095247_20240121T113418_32504_03_020600_20240123T022011.nc': RasterioIOError('HTTP response code: 404')
warnings.warn(msg)
c:\Users\fuji_\anaconda3\envs\geo\Lib\site-packages\stackstac\rio_reader.py:329: UserWarning: Error opening 'https://sentinel5euwest.blob.core.windows.net/sentinel-5p/TROPOMI/L2__CH4___/2024/07/24/S5P_OFFL_L2__CH4____20240724T103609_20240724T121739_35129_03_020600_20240726T072607/S5P_OFFL_L2__CH4____20240724T103609_20240724T121739_35129_03_020600_20240726T072607.nc': RasterioIOError('HTTP response code: 404')
warnings.warn(msg)
Output is truncated. View as a scrollable element or open in a text editor. Adjust cell output settings...
[################################## ] | 85% Completed | 1.96 s
c:\Users\fuji_\anaconda3\envs\geo\Lib\site-packages\stackstac\rio_reader.py:329: UserWarning: Error opening 'https://sentinel5euwest.blob.core.windows.net/sentinel-5p/TROPOMI/L2__CH4___/2024/07/29/S5P_OFFL_L2__CH4____20240729T104210_20240729T122340_35200_03_020600_20240804T092141/S5P_OFFL_L2__CH4____20240729T104210_20240729T122340_35200_03_020600_20240804T092141.nc': RasterioIOError('HTTP response code: 404')
warnings.warn(msg)
c:\Users\fuji_\anaconda3\envs\geo\Lib\site-packages\stackstac\rio_reader.py:329: UserWarning: Error opening 'https://sentinel5euwest.blob.core.windows.net/sentinel-5p/TROPOMI/L2__CH4___/2024/08/03/S5P_OFFL_L2__CH4____20240803T104809_20240803T122938_35271_03_020600_20240809T130432/S5P_OFFL_L2__CH4____20240803T104809_20240803T122938_35271_03_020600_20240809T130432.nc': RasterioIOError('HTTP response code: 404')
warnings.warn(msg)
c:\Users\fuji_\anaconda3\envs\geo\Lib\site-packages\stackstac\rio_reader.py:329: UserWarning: Error opening 'https://sentinel5euwest.blob.core.windows.net/sentinel-5p/TROPOMI/L2__CH4___/2024/08/08/S5P_OFFL_L2__CH4____20240808T105404_20240808T123534_35342_03_020600_20240812T214910/S5P_OFFL_L2__CH4____20240808T105404_20240808T123534_35342_03_020600_20240812T214910.nc': RasterioIOError('HTTP response code: 404')
warnings.warn(msg)
c:\Users\fuji_\anaconda3\envs\geo\Lib\site-packages\stackstac\rio_reader.py:329: UserWarning: Error opening 'https://sentinel5euwest.blob.core.windows.net/sentinel-5p/TROPOMI/L2__CH4___/2024/08/18/S5P_OFFL_L2__CH4____20240818T110547_20240818T124717_35484_03_020600_20240820T031012/S5P_OFFL_L2__CH4____20240818T110547_20240818T124717_35484_03_020600_20240820T031012.nc': RasterioIOError('HTTP response code: 404')
warnings.warn(msg)
c:\Users\fuji_\anaconda3\envs\geo\Lib\site-packages\stackstac\rio_reader.py:329: UserWarning: Error opening 'https://sentinel5euwest.blob.core.windows.net/sentinel-5p/TROPOMI/L2__CH4___/2024/08/23/S5P_OFFL_L2__CH4____20240823T111134_20240823T125304_35555_03_020600_20240825T033549/S5P_OFFL_L2__CH4____20240823T111134_20240823T125304_35555_03_020600_20240825T033549.nc': RasterioIOError('HTTP response code: 404')
warnings.warn(msg)
c:\Users\fuji_\anaconda3\envs\geo\Lib\site-packages\stackstac\rio_reader.py:329: UserWarning: Error opening 'https://sentinel5euwest.blob.core.windows.net/sentinel-5p/TROPOMI/L2__CH4___/2024/08/28/S5P_OFFL_L2__CH4____20240828T111719_20240828T125848_35626_03_020600_20240830T032246/S5P_OFFL_L2__CH4____20240828T111719_20240828T125848_35626_03_020600_20240830T032246.nc': RasterioIOError('HTTP response code: 404')
warnings.warn(msg)
c:\Users\fuji_\anaconda3\envs\geo\Lib\site-packages\stackstac\rio_reader.py:329: UserWarning: Error opening 'https://sentinel5euwest.blob.core.windows.net/sentinel-5p/TROPOMI/L2__CH4___/2024/09/02/S5P_OFFL_L2__CH4____20240902T112301_20240902T130431_35697_03_020600_20240904T034141/S5P_OFFL_L2__CH4____20240902T112301_20240902T130431_35697_03_020600_20240904T034141.nc': RasterioIOError('HTTP response code: 404')
warnings.warn(msg)
c:\Users\fuji_\anaconda3\envs\geo\Lib\site-packages\stackstac\rio_reader.py:329: UserWarning: Error opening 'https://sentinel5euwest.blob.core.windows.net/sentinel-5p/TROPOMI/L2__CH4___/2024/09/07/S5P_OFFL_L2__CH4____20240907T112851_20240907T131021_35768_03_020600_20240909T040338/S5P_OFFL_L2__CH4____20240907T112851_20240907T131021_35768_03_020600_20240909T040338.nc': RasterioIOError('HTTP response code: 404')
warnings.warn(msg)
c:\Users\fuji_\anaconda3\envs\geo\Lib\site-packages\stackstac\rio_reader.py:329: UserWarning: Error opening 'https://sentinel5euwest.blob.core.windows.net/sentinel-5p/TROPOMI/L2__CH4___/2024/09/17/S5P_OFFL_L2__CH4____20240917T114040_20240917T132210_35910_03_020701_20240919T063955/S5P_OFFL_L2__CH4____20240917T114040_20240917T132210_35910_03_020701_20240919T063955.nc': RasterioIOError('HTTP response code: 404')
warnings.warn(msg)
c:\Users\fuji_\anaconda3\envs\geo\Lib\site-packages\stackstac\rio_reader.py:329: UserWarning: Error opening 'https://sentinel5euwest.blob.core.windows.net/sentinel-5p/TROPOMI/L2__CH4___/2024/08/13/S5P_OFFL_L2__CH4____20240813T105958_20240813T124127_35413_03_020600_20240815T215037/S5P_OFFL_L2__CH4____20240813T105958_20240813T124127_35413_03_020600_20240815T215037.nc': RasterioIOError('HTTP response code: 404')
warnings.warn(msg)
c:\Users\fuji_\anaconda3\envs\geo\Lib\site-packages\stackstac\rio_reader.py:329: UserWarning: Error opening 'https://sentinel5euwest.blob.core.windows.net/sentinel-5p/TROPOMI/L2__CH4___/2024/09/22/S5P_OFFL_L2__CH4____20240922T114632_20240922T132801_35981_03_020701_20240924T071754/S5P_OFFL_L2__CH4____20240922T114632_20240922T132801_35981_03_020701_20240924T071754.nc': RasterioIOError('HTTP response code: 404')
warnings.warn(msg)
c:\Users\fuji_\anaconda3\envs\geo\Lib\site-packages\stackstac\rio_reader.py:329: UserWarning: Error opening 'https://sentinel5euwest.blob.core.windows.net/sentinel-5p/TROPOMI/L2__CH4___/2024/09/12/S5P_OFFL_L2__CH4____20240912T113447_20240912T131617_35839_03_020701_20240914T201419/S5P_OFFL_L2__CH4____20240912T113447_20240912T131617_35839_03_020701_20240914T201419.nc': RasterioIOError('HTTP response code: 404')
warnings.warn(msg)
c:\Users\fuji_\anaconda3\envs\geo\Lib\site-packages\stackstac\rio_reader.py:329: UserWarning: Error opening 'https://sentinel5euwest.blob.core.windows.net/sentinel-5p/TROPOMI/L2__CH4___/2024/01/26/S5P_OFFL_L2__CH4____20240126T114058_20240126T132228_32576_03_020600_20240128T051841/S5P_OFFL_L2__CH4____20240126T114058_20240126T132228_32576_03_020600_20240128T051841.nc': RasterioIOError('HTTP response code: 404')
...
c:\Users\fuji_\anaconda3\envs\geo\Lib\site-packages\stackstac\rio_reader.py:329: UserWarning: Error opening 'https://sentinel5euwest.blob.core.windows.net/sentinel-5p/TROPOMI/L2__CH4___/2024/11/12/S5P_OFFL_L2__CH4____20241112T104321_20241112T122450_36704_03_020701_20241115T183324/S5P_OFFL_L2__CH4____20241112T104321_20241112T122450_36704_03_020701_20241115T183324.nc': RasterioIOError('HTTP response code: 404')
warnings.warn(msg)
c:\Users\fuji_\anaconda3\envs\geo\Lib\site-packages\stackstac\rio_reader.py:329: UserWarning: Error opening 'https://sentinel5euwest.blob.core.windows.net/sentinel-5p/TROPOMI/L2__CH4___/2024/12/17/S5P_OFFL_L2__CH4____20241217T094534_20241217T112704_37200_03_020800_20241219T020255/S5P_OFFL_L2__CH4____20241217T094534_20241217T112704_37200_03_020800_20241219T020255.nc': RasterioIOError('HTTP response code: 404')
warnings.warn(msg)
Output is truncated. View as a scrollable element or open in a text editor. Adjust cell output settings...
[########################################] | 100% Completed | 2.06 s
c:\Users\fuji_\anaconda3\envs\geo\Lib\site-packages\stackstac\rio_reader.py:329: UserWarning: Error opening 'https://sentinel5euwest.blob.core.windows.net/sentinel-5p/TROPOMI/L2__CH4___/2024/02/05/S5P_OFFL_L2__CH4____20240205T115424_20240205T133554_32718_03_020600_20240207T042306/S5P_OFFL_L2__CH4____20240205T115424_20240205T133554_32718_03_020600_20240207T042306.nc': RasterioIOError('HTTP response code: 404')
warnings.warn(msg)
c:\Users\fuji_\anaconda3\envs\geo\Lib\site-packages\stackstac\rio_reader.py:329: UserWarning: Error opening 'https://sentinel5euwest.blob.core.windows.net/sentinel-5p/TROPOMI/L2__CH4___/2024/02/10/S5P_OFFL_L2__CH4____20240210T120110_20240210T134240_32789_03_020600_20240212T042117/S5P_OFFL_L2__CH4____20240210T120110_20240210T134240_32789_03_020600_20240212T042117.nc': RasterioIOError('HTTP response code: 404')
warnings.warn(msg)
c:\Users\fuji_\anaconda3\envs\geo\Lib\site-packages\stackstac\rio_reader.py:329: UserWarning: Error opening 'https://sentinel5euwest.blob.core.windows.net/sentinel-5p/TROPOMI/L2__CH4___/2024/02/16/S5P_OFFL_L2__CH4____20240216T100729_20240216T114900_32873_03_020600_20240218T023150/S5P_OFFL_L2__CH4____20240216T100729_20240216T114900_32873_03_020600_20240218T023150.nc': RasterioIOError('HTTP response code: 404')
warnings.warn(msg)

And when i try to save the datarray by computing the mean over this clipped area, i get a tiff file with no results

Comece pelo código do notebook que pesquisa a Planetary Computer STAC API e passa os assets ch4 para stackstac.stack; inspecione as URLs de blob do Sentinel-5P geradas e as respostas 404 relatadas. Confirme se os links dos assets STAC são válidos e identifique o comportamento esperado quando o stack é persistido e recortado sem erros ao abrir os assets.