isce-framework / isce-framework/dolphin

S3 file support

Open
#173 6 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
153
Forks
33
PR merge metrics
No merged PRs in 30d

Description

Opening a meta issue for possible changes/additions to better support interacting with files on S3

  • We might need something like s3path (but smaller) to interact with the filenames... using pathlib.Path('s3://my-bucket') leaves you with Path(s3:/my-bucket).
  • rasterio has built-in support for strings like s3://my-bucket/my-file.tif, while using gdal bindings requires you to convert it to /vsis3/my-bucket/my-file.tif. That sounds annoying enough to find all edge cases that it might be worth the slow switchover to rasterio
  • Probably a separate bigger issue related to the reader issue, but it would be useful to have hdf5 streaming capability

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by separating the three proposals in the issue: S3 path handling, rasterio versus GDAL S3 paths, and HDF5 streaming. Read the linked s3path repository and HDF5 gist, then compare the rasterio and GDAL approaches; done requires a maintainer-approved scope rather than this broad meta issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud, data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.