isce-framework / isce-framework/dolphin
S3 file support
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 153
- Forks
- 33
- PR merge metrics
- No merged PRs in 30d
Description
Opening a meta issue for possible changes/additions to better support interacting with files on S3
- We might need something like s3path (but smaller) to interact with the filenames... using
pathlib.Path('s3://my-bucket')leaves you withPath(s3:/my-bucket). rasteriohas built-in support for strings likes3://my-bucket/my-file.tif, while usinggdalbindings requires you to convert it to/vsis3/my-bucket/my-file.tif. That sounds annoying enough to find all edge cases that it might be worth the slow switchover to rasterio- Probably a separate bigger issue related to the reader issue, but it would be useful to have hdf5 streaming capability
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by separating the three proposals in the issue: S3 path handling, rasterio versus GDAL S3 paths, and HDF5 streaming. Read the linked s3path repository and HDF5 gist, then compare the rasterio and GDAL approaches; done requires a maintainer-approved scope rather than this broad meta issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cloud, data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100