codecombat / codecombat/codecombat

Easy reproduction for memory issues in Python

Open
#3,343 3 comments 0 reactions 1 assignee View on GitHub

@basicer is already working on this.

Since May 22, 2016.

Dominant language
JavaScript
Stars
8.6k
Forks
4.2k
Avg merge
1d 4h
Merged PRs (30d)
39

Description

Produced in Firefox. Load Multiplayer Treasure hunt and use the following code (Note this level has a lot of objects which probably helps reproduce this faster).

while True:
    x = 0
    for i in range(0,50000):
        x = x + 2
    self.shield()

You will hit the execution limit at 27.5333
Change to

while True:
    x = 0
    for i in range(0,50000):
        x += 2
    self.shield()

You will not hit the execution limit at all when using += here. (!! That means this is more than 4x faster!)

Change again to

while True:
    x = 0
    for i in range(0,100000):
        x += 2
    self.shield()

You would think that you are only doubling the number of steps, but clearly you are transpiling into python 2.x, because actually this will hit the limit before 60 seconds (if it was only doing twice the work, the soonest it could hit the limit would be 60 seconds in a 120 second map). Thus it's probably allocating a list of length 100k when it does this loop.

There are plenty of weird things you can notice when you do this (subtracting is the same speed as divide), but that's neither here nor there.

If you repeatedly make minor edits, such as changing it to +1, -1, *= 3, etc. Waiting for the full simulation isn't necessary, just wait for the transpile to finish and it to start running, though waiting a few seconds can't hurt. Monitor the memory use of your browser as you do this. You will easily gain a few hundred megabytes to gigabyte after a few dozen tries. Eventually you will OOM and crash firefox.

Conclusion:
The first version is allocating an object for x at each iteration of the loop, but these objects are not getting cleaned up. They both are using the same range, so the difference must be doing much more work than expected.

I managed to get log messages before the browser crashed on my reproduction attempt:

嫦娥: Let there be light upon Multiplayer Treasure Grove! (preload: false) multiplayer-treasure-grove:398:1
|嫦娥's Ray| Generated random seed -690272733 of type playerSession from sessionIDs 567f0790a658472000437f97 submissionCount 7 multiplayer-treasure-grove:43:10
Automatically scrolling cursor into view after selection change this will be disabled in the next version set editor.$blockScrolling = Infinity to disable this message ace.js:1:51268
unreachable code after return statement multiplayer-treasure-grove:2453:0
|嫦娥's Ray|   Loaded 23 of 1800 (+1007ms) multiplayer-treasure-grove:43:10
|嫦娥's Ray| Hero Placeholder had new Programmable problem: plan Hard execution limit of 3000000 exceeded. 4.2 multiplayer-treasure-grove:43:10
|嫦娥's Ray|   Loaded 65 of 1800 (+1171ms) multiplayer-treasure-grove:43:10
|嫦娥's Ray|   Loaded 1416 of 1800 (+1083ms) multiplayer-treasure-grove:43:10
|嫦娥's Ray| And it was so: (2.242ms per frame, 1800 frames)
Simulation   : 4036ms 
Serialization: 30ms
Delivery     : 8ms multiplayer-treasure-grove:43:10
First changed frame is 63 with hash 487930846.33347803 compared to 487930844.69307375 multiplayer-treasure-grove:6443:62
Automatically scrolling cursor into view after selection change this will be disabled in the next version set editor.$blockScrolling = Infinity to disable this message ace.js:1:51268
going to log an error message app.js:700:1
Error saving LevelSession undefined app.js:701:1
Couldn't even show noty error for undefined because out of memory app.js:703:1
嫦娥: Let there be light upon Multiplayer Treasure Grove! (preload: false) multiplayer-treasure-grove:398:1
|嫦娥's Ray| Generated random seed -690272733 of type playerSession from sessionIDs 567f0790a658472000437f97 submissionCount 7 multiplayer-treasure-grove:43:10
unreachable code after return statement multiplayer-treasure-grove:2453:0
|嫦娥's Ray|   Loaded 163 of 1800 (+1020ms) multiplayer-treasure-grove:43:10
|嫦娥's Ray| Hero Placeholder had new Programmable problem: plan Hard execution limit of 3000000 exceeded. 13.533333333333333 multiplayer-treasure-grove:43:10
|嫦娥's Ray|   Loaded 1145 of 1800 (+1023ms) multiplayer-treasure-grove:43:10
|嫦娥's Ray| And it was so: (1.709ms per frame, 1800 frames)
Simulation   : 3076ms 
Serialization: 30ms
Delivery     : 7ms multiplayer-treasure-grove:43:10
First changed frame is 63 with hash 487930844.69307375 compared to 487930846.33347803 multiplayer-treasure-grove:6443:62
going to log an error message app.js:700:1
Error saving LevelSession undefined app.js:701:1
Couldn't even show noty error for undefined because Exception { message: "", result: 2147942414, name: "NS_ERROR_OUT_OF_MEMORY", filename: "http://codecombat.com/javascripts/v…", lineNumber: 585, columnNumber: 0, inner: null, data: null, stack: ".buildFragment@http://codecombat.co…" } app.js:703:1
嫦娥: Let there be light upon Multiplayer Treasure Grove! (preload: false) multiplayer-treasure-grove:398:1
|嫦娥's Ray| Generated random seed -690272733 of type playerSession from sessionIDs 567f0790a658472000437f97 submissionCount 7 multiplayer-treasure-grove:43:10
unreachable code after return statement multiplayer-treasure-grove:2453:0
|嫦娥's Ray|   Loaded 79 of 1800 (+1053ms) multiplayer-treasure-grove:43:10
|嫦娥's Ray| Hero Placeholder had new Programmable problem: plan out of memory 10.266666666666666 multiplayer-treasure-grove:43:10
|嫦娥's Ray| World delivery error: out of memory
undefined multiplayer-treasure-grove:43:10
|嫦娥's Ray|   Loaded 483 of 1800 (+1014ms) multiplayer-treasure-grove:43:10
|嫦娥's Ray| World delivery error: out of memory
undefined multiplayer-treasure-grove:43:10
|嫦娥's Ray|   Loaded 1618 of 1800 (+1091ms) multiplayer-treasure-grove:43:10
|嫦娥's Ray| World delivery error: out of memory
undefined multiplayer-treasure-grove:43:10
|嫦娥's Ray| And it was so: (1.929ms per frame, 1800 frames)
Simulation   : 3472ms 
Serialization: 54ms
Delivery     : 3ms multiplayer-treasure-grove:43:10
going to log an error message app.js:700:1
Error saving LevelSession undefined app.js:701:1
Couldn't even show noty error for undefined because out of memory app.js:703:1
Could not write session state file  out of memory undefined SessionFile.jsm:291
Automatically scrolling cursor into view after selection change this will be disabled in the next version set editor.$blockScrolling = Infinity to disable this message ace.js:1:51268
uncaught exception: out of memory <unknown>
Could not write session state file  out of memory undefined SessionFile.jsm:291
out of memory <unknown>
uncaught exception: out of memory <unknown>
out of memory <unknown>
Could not write session state file  out of memory undefined SessionFile.jsm:291
uncaught exception: out of memory <unknown>
Could not write session state file  out of memory undefined SessionFile.jsm:291
out of memory <unknown>
Could not write session state file  out of memory undefined

The app hung when the first OOM errors started getting logged, resulting in all coins and player sprites to vanish. After about a minute, the coins and player reappeared, and more OOM errors were added continually every couple of seconds.

With this info, it should be easy to reproduce and track down.

After 10 minutes of idling (not running more iterations so my browser doesn't straight up crash again) something like this started to appear every few seconds:

"Could not write session state file  out of memory undefined SessionFile.jsm:291
out of memory <unknown>
Could not write session state file  Error: TypeError: invalid 'in' operand exn
Stack trace:
postMessage@resource://gre/modules/PromiseWorker.jsm:324:1
TaskImpl_run@resource://gre/modules/Task.jsm:315:40
Handler.prototype.process@resource://gre/modules/Promise.jsm -> resource://gre/modules/Promise-backend.js:937:21
this.PromiseWalker.walkerLoop@resource://gre/modules/Promise.jsm -> resource://gre/modules/Promise-backend.js:813:7
this.PromiseWalker.scheduleWalkerLoop/<@resource://gre/modules/Promise.jsm -> resource://gre/modules/Promise-backend.js:747:1
 "postMessage@resource://gre/modules/PromiseWorker.jsm:324:1
TaskImpl_run@resource://gre/modules/Task.jsm:315:40
Handler.prototype.process@resource://gre/modules/Promise.jsm -> resource://gre/modules/Promise-backend.js:937:21
this.PromiseWalker.walkerLoop@resource://gre/modules/Promise.jsm -> resource://gre/modules/Promise-backend.js:813:7
this.PromiseWalker.scheduleWalkerLoop/<@resource://gre/modules/Promise.jsm -> resource://gre/modules/Promise-backend.js:747:1
"

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.