parse-community / parse-community/parse-server

Fetch large data for reports slow

Open
#6,543 67 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

type:feature
Dominant language
JavaScript
Stars
21.4k
Forks
4.8k
Avg merge
7h 45m
Merged PRs (30d)
11

Description

I'm trying to fetch more than 100,000 rows for report generation.
I tried these approaches,

  1. Recursive calls with 1,000 limit to fetch all rows
  2. Using query.each() and push row in array

Both the calls are slow,

For cloud code query

query = //new query
query.equalTo("SOMEKEY","SOMEVALUE")

For total of 1,500,000 rows
In cloud code,

In recursive call approach, there is approx 1 second waiting time per 1000 rows.

In query.each() approach this query takes approx 60 seconds to process, http waiting time

I assume this is fastest parse can work.

Can i see some improvement by indexing the mongodb objectId column?

Is there a better approach for processing such huge data.

One alternative I can think of is parse aggregate but this approach doesn't respect parse authentication, security, acl etc. and doesn't work with existing query logic.
Will this show any performance improvement and worth a try?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue names Cloud Code query usage, query.each(), recursive 1,000-row calls, MongoDB objectId indexing, and Parse aggregate. Start by comparing those approaches against the existing query logic and its authentication and ACL requirements; a concrete code location, test, and acceptance criterion are not specified.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, mongodb, nodejs
Domain
backend, databases, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.