developmentseed / developmentseed/pgstac-ingestor
Allow for an ingestion of more than one item
- Dominant language
- Python
- Stars
- 7
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
Currently the API only accepts single items https://github.com/developmentseed/pgstac-ingestor/blob/38a2a55c7bd255684383550844b8e40323b97f69/pgstac_ingestor/main.py#L32.
For scenarios where one wants to ingest in one shot a large number of items in bulk (say, 500k items stored in an ndjson at once), but where a direct connection to the DB (and therefore simple usage of `pypgstac`) is unacceptable, it's quite inefficient to use the API to insert one by one all the items.
It would be desirable to expose an API for one-shot large ingestion tasks.
Thoughts @sharkinsspatial @vincentsarago ?
Contributor guide
Research direction
Start with the existing single-item API in pgstac_ingestor/main.py at line 32. The issue proposes a bulk-ingestion API for large sets of items, such as an NDJSON file; clarify the API design and expected behavior before deciding what would count as done.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgresql, python
- Domain
- api, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100