parse-community / parse-community/parse-server
Fetch large data for reports slow
Nobody has claimed this yet.
- Dominant language
- JavaScript
- Stars
- 21.4k
- Forks
- 4.8k
- Avg merge
- 7h 45m
- Merged PRs (30d)
- 11
Description
I'm trying to fetch more than 100,000 rows for report generation.
I tried these approaches,
- Recursive calls with 1,000 limit to fetch all rows
- Using query.each() and push row in array
Both the calls are slow,
For cloud code query
query = //new query
query.equalTo("SOMEKEY","SOMEVALUE")
For total of 1,500,000 rows
In cloud code,
In recursive call approach, there is approx 1 second waiting time per 1000 rows.
In query.each() approach this query takes approx 60 seconds to process, http waiting time
I assume this is fastest parse can work.
Can i see some improvement by indexing the mongodb objectId column?
Is there a better approach for processing such huge data.
One alternative I can think of is parse aggregate but this approach doesn't respect parse authentication, security, acl etc. and doesn't work with existing query logic.
Will this show any performance improvement and worth a try?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue names Cloud Code query usage, query.each(), recursive 1,000-row calls, MongoDB objectId indexing, and Parse aggregate. Start by comparing those approaches against the existing query logic and its authentication and ACL requirements; a concrete code location, test, and acceptance criterion are not specified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, mongodb, nodejs
- Domain
- backend, databases, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100