parse-community / parse-community/parse-server

Huge log files created and crashes the app after API scriptlog call

Open
#6,420 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

type:feature
Dominant language
JavaScript
Stars
21.4k
Forks
4.8k
Avg merge
7h 45m
Merged PRs (30d)
11

Description

Issue Description

Hi
My log file grow to 50MB and more for a single file.
A. how to prevent this from happening? (lets assume i want to continue verbose logs and thats the amount of data that i have)
B. Once going to the log section on the dashboard (or doing the API call, thats why its not a dashboard issue), my server CPU jumped high, node memory went over 2GB (which crashed my AWS t3.small server)

Steps to reproduce

Not an easy thing to do, unfortunately i can't share my file with you because of security details inside the log.
My data structure has something like a 1 MB data in each document (row) of a specific collection.
Calling the Class Get all many times will start to fill the log file.
Total log library size was 150MB (100+ files) with 2 very large files:
60MB of info.log and 40MB of error.log of 1 specific day.
Once you have that files call the API XXX/parse/scriptlog?level=info&n=100
eventually the call either return 504 or crash the server

Expected Results

Max log file size should be 5MB(configurable?)
Query logs should not crash or consume all memory.

Actual Outcome

60MB log file and crash the system

Environment Setup
  • Server

    • parse-server version (Be specific! Don't say 'latest'.) : 3.10.0
    • Operating System: AWS linux and Mac Os Catalina
    • Hardware: t3.small which is 2CPU 2GB RAM
    • Localhost or remote server? (AWS, Heroku, Azure, Digital Ocean, etc): AWS (happens on localhost as well)
  • Database

    • MongoDB version: 3.4.0
    • Storage engine: [FILL THIS OUT]
    • Hardware: 4 cpu 8GB RAM
    • Localhost or remote server? (AWS, mLab, ObjectRocket, Digital Ocean, etc): AWS EC2
Logs/Trace

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by examining the /parse/scriptlog?level=info&n=100 entry point and reproduce the request with the large info.log and error.log files described in the issue. Done means verbose logging can continue without unbounded log files, and querying the logs no longer causes excessive memory use, a 504, or a server crash.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, node.js
Domain
api, backend, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.