NASA-IMPACT / NASA-IMPACT/QuARC
Scaling quarc API
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
Limitations:
- Currrently quarc api is not able to process large chunks of metadata at a time. It's because of the limitations of the AWS API gateway (can only run for 29 s) and the lambda function (can only run for 15 min).
Discussions
Some alternatives that we have come up so far:
- Spin up multiple lambda functions where each lambda function has the same image of pyquarc package that could handle single collection and coalesce them to return as a single response to the user.
- Or Each sequential step of pyquarc package like getting list of concept ids, downloading the metadata, processing the metadata could be distributed across multiple lambda functions that are orchestrated by step functions
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue names no files or tests. Start by tracing the pyQuARC API flow and reviewing the AWS API Gateway and Lambda limits described; evaluate the listed Lambda-parallelism and Step Functions alternatives. A completion criterion is not specified, so scope and validation need maintainer clarification.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- api, backend, cloud
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100