Indexer crashes: FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory

The indexer crashes with an out of memory error. Also described here.

<--- Last few GCs --->

[821998:0x5b21020]   521524 ms: Mark-sweep 2023.1 (2089.5) -> 2011.8 (2094.0) MB, 1277.7 / 0.0 ms  (average mu = 0.383, current mu = 0.035) allocation failure; scavenge might not succeed
[821998:0x5b21020]   523403 ms: Mark-sweep 2027.7 (2094.0) -> 2016.5 (2098.5) MB, 1840.4 / 0.0 ms  (average mu = 0.204, current mu = 0.021) allocation failure; scavenge might not succeed


<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
 1: 0xb6e260 node::Abort() [/home/test/.nvm/versions/node/v18.8.0/bin/node]
 2: 0xa7cdea  [/home/test/.nvm/versions/node/v18.8.0/bin/node]
 3: 0xd45f00 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/home/test/.nvm/versions/node/v18.8.0/bin/node]
 4: 0xd462a7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/home/test/.nvm/versions/node/v18.8.0/bin/node]
 5: 0xf235b5  [/home/test/.nvm/versions/node/v18.8.0/bin/node]
 6: 0xf35a9d v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/home/test/.nvm/versions/node/v18.8.0/bin/node]
 7: 0xf1019e v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/home/test/.nvm/versions/node/v18.8.0/bin/node]
 8: 0xf11567 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/home/test/.nvm/versions/node/v18.8.0/bin/node]
 9: 0xef273a v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/home/test/.nvm/versions/node/v18.8.0/bin/node]
10: 0x12b5c4f v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/home/test/.nvm/versions/node/v18.8.0/bin/node]
11: 0x16e78b9  [/home/test/.nvm/versions/node/v18.8.0/bin/node]

To reproduce

# in one terminal start the indexer
packages/cli$ yarn start index --watch --appmap-dir ~/src/sample_app_6th_ed/tmp/appmap/minitest

# in another terminal keep creating and deleting appmaps for a little over 5 minutes.
~/src/sample_app_6th_ed$ rm tmp/appmap/minitest/*; APPMAP=true DISABLE_SPRING=true bundle exec rails test; sleep 60; rm tmp/appmap/minitest/*; APPMAP=true DISABLE_SPRING=true bundle exec rails test; sleep 60; rm tmp/appmap/minitest/*; APPMAP=true DISABLE_SPRING=true bundle exec rails test; sleep 60; rm tmp/appmap/minitest/*; APPMAP=true DISABLE_SPRING=true bundle exec rails test; sleep 60; rm tmp/appmap/minitest/*; APPMAP=true DISABLE_SPRING=true bundle exec rails test; sleep 60; rm tmp/appmap/minitest/*; APPMAP=true DISABLE_SPRING=true bundle exec rails test; sleep 60; rm tmp/appmap/minitest/*; APPMAP=true DISABLE_SPRING=true bundle exec rails test; sleep 60; rm tmp/appmap/minitest/*; APPMAP=true DISABLE_SPRING=true bundle exec rails test; sleep 60; rm tmp/appmap/minitest/*; APPMAP=true DISABLE_SPRING=true bundle exec rails test; sleep 60; rm tmp/appmap/minitest/*; APPMAP=true DISABLE_SPRING=true bundle exec rails test; sleep 60;

Also reproduced with sleep 30 instead of sleep 60.

kgilpin suggests

The first thing I would check is that the indexer uses a single queue
across all the create/update events. It's probably running a separate queue
for each event and so with many create/update events, it can eventually run
out of memory with all that AppMap data concurrently loaded.

Same for the scanner.