developmentseed / developmentseed/pgstac-ingestor

Allow for an ingestion of more than one item

Open
#10 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
7
Forks
0
PR merge metrics
No merged PRs in 30d

Description

Currently the API only accepts single items https://github.com/developmentseed/pgstac-ingestor/blob/38a2a55c7bd255684383550844b8e40323b97f69/pgstac_ingestor/main.py#L32.

For scenarios where one wants to ingest in one shot a large number of items in bulk (say, 500k items stored in an ndjson at once), but where a direct connection to the DB (and therefore simple usage of `pypgstac`) is unacceptable, it's quite inefficient to use the API to insert one by one all the items.

It would be desirable to expose an API for one-shot large ingestion tasks.

Thoughts @sharkinsspatial @vincentsarago ?

Contributor guide

Open the contributing guide

Research direction

Start with the existing single-item API in pgstac_ingestor/main.py at line 32. The issue proposes a bulk-ingestion API for large sets of items, such as an NDJSON file; clarify the API design and expected behavior before deciding what would count as done.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, python
Domain
api, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.