HaxeFoundation / HaxeFoundation/haxe

haxe.Serializer encounters Stack Overflow when processing 31 element TEnum on cpp

Open
#12,009 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Haxe
Stars
6.9k
Forks
715
Avg merge
2d 2h
Merged PRs (30d)
11

Description

Dev Environment: Windows
Build command: lime test windows -debug

haxe 4.3.6
lime 8.1.2
openfl 9.3.3
haxeflixel 5.8.0
hxcpp-debugger 1.1

I'm trying to use the hxcpp debugger and the application keeps crashing due to a stack overflow in haxe.Serialize.

Here is what's being passed to Serialize.run()

  | Name | Value | Type
-- | -- | -- | --
◢ | __vfptr | 0x00007ff608685428 {FlxProject.exe!void(* debugger::Message_obj::`vftable'[34])()} {0x00007ff60600664e {FlxProject.exe!hx::EnumBase_obj::_hx_isInstanceOf(int)}, ...} | void * *
  | [0] | 0x00007ff60600664e {FlxProject.exe!hx::EnumBase_obj::_hx_isInstanceOf(int)} | void *
  | [1] | 0x00007ff6060077f1 {FlxProject.exe!hx::EnumBase_obj::__Mark(class hx::MarkContext *)} | void *
  | [2] | 0x00007ff606009632 {FlxProject.exe!hx::EnumBase_obj::__Visit(class hx::VisitContext *)} | void *
  | [3] | 0x00007ff605ff1a6e {FlxProject.exe!hx::EnumBase_obj::__GetType(void)const } | void *
  | [4] | 0x00007ff6060145fa {FlxProject.exe!hx::Object::__GetHandle(void)const } | void *
  | [5] | 0x00007ff605ffd238 {FlxProject.exe!hx::Object::__FieldRef(class String const &)} | void *
  | [6] | 0x00007ff60602342e {FlxProject.exe!debugger::Message_obj::__ToString(void)const } | void *
  | [7] | 0x00007ff605ff7b8f {FlxProject.exe!hx::Object::__ToInt(void)const } | void *
  | [8] | 0x00007ff60600e777 {FlxProject.exe!hx::Object::__ToDouble(void)const } | void *
  | [9] | 0x00007ff60602170f {FlxProject.exe!hx::Object::__ToInt64(void)const } | void *
  | [10] | 0x00007ff606036a51 {FlxProject.exe!hx::Object::__CStr(void)const } | void *
  | [11] | 0x00007ff605ff1320 {FlxProject.exe!hx::EnumBase_obj::toString(void)} | void *
  | [12] | 0x00007ff60601a28e {FlxProject.exe!hx::Object::__HasField(class String const &)} | void *
  | [13] | 0x00007ff606017a93 {FlxProject.exe!debugger::Message_obj::__Field(class String const &,enum hx::PropertyAccessMode)} | void *
  | [14] | 0x00007ff6060262a5 {FlxProject.exe!hx::Object::_hx_getInterface(int)} | void *
  | [15] | 0x00007ff606033f04 {FlxProject.exe!hx::Object::__SetField(class String const &,struct cpp::Variant const &,enum hx::PropertyAccessMode)} | void *
  | [16] | 0x00007ff606038266 {FlxProject.exe!hx::Object::__SetThis(class Dynamic)} | void *
  | [17] | 0x00007ff605ff18ac {FlxProject.exe!hx::Object::__Run(class Array const &)} | void *
  | [18] | 0x00007ff60602a125 {FlxProject.exe!hx::Object::__GetFieldMap(void)} | void *
  | [19] | 0x00007ff6060135c4 {FlxProject.exe!hx::Object::__GetFields(class Array &)} | void *
  | [20] | 0x00007ff605ffc2ca {FlxProject.exe!debugger::Message_obj::__GetClass(void)const } | void *
  | [21] | 0x00007ff6060213e5 {FlxProject.exe!hx::EnumBase_obj::__Compare(class hx::Object const *)const } | void *
  | [22] | 0x00007ff606013dd0 {FlxProject.exe!hx::Object::__length(void)const } | void *
  | [23] | 0x00007ff606015054 {FlxProject.exe!hx::EnumBase_obj::__GetItem(int)const } | void *
  | [24] | 0x00007ff606033946 {FlxProject.exe!hx::Object::__SetItem(int,class Dynamic)} | void *
  | [25] | 0x00007ff60600fe51 {FlxProject.exe!hx::Object::__run(class Dynamic const &,class Dynamic const &,class Dynamic const &,class Dynamic const &,class Dynamic const &)} | void *
  | [26] | 0x00007ff6060269ee {FlxProject.exe!hx::Object::__run(class Dynamic const &,class Dynamic const &,class Dynamic const &,class Dynamic const &)} | void *
  | [27] | 0x00007ff60602305f {FlxProject.exe!hx::Object::__run(class Dynamic const &,class Dynamic const &,class Dynamic const &)} | void *
  | [28] | 0x00007ff605ff96e7 {FlxProject.exe!hx::Object::__run(class Dynamic const &,class Dynamic const &)} | void *
  | [29] | 0x00007ff606027b64 {FlxProject.exe!hx::Object::__run(class Dynamic const &)} | void *
  | [30] | 0x00007ff60602d433 {FlxProject.exe!hx::Object::__run(void)} | void *
  | [31] | 0x00007ff606020c15 {FlxProject.exe!hx::Object::__ArgCount(void)const } | void *

Here's the stack trace.
Unhandled exception at 0x00007FF6080AA4D7 in FlxProject.exe: 0xC00000FD: Stack overflow (parameters: 0x0000000000000001, 0x00000052A5003000).

> FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 387 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::serialize(Dynamic v) Line 1398 C++
FlxProject.exe!haxe::Serializer_obj::run(Dynamic v) Line 1430 C++
FlxProject.exe!debugger::DebuggerThread_obj::debuggerThreadMain() Line 432 C++

debugger.DebuggerThread that contains the Serialize.run() call looks like this, I tried to do some tracing or serializing to generate a reproducible sample but I didn't have much luck.
```
case GetStructured(unsafe, expression):
trace(expression);
var message = this.getStructured(unsafe, expression);
var json = haxe.Json.stringify(message);
trace("JSON INCOMING");
trace("");
trace(json);
trace("");
trace("Stopped?");
trace("");
var serialized = haxe.Serializer.run(message);
trace(serialized);
emit(message);
```

```
debugger/DebuggerThread.hx:254: this
debugger/DebuggerThread.hx:257: JSON INCOMING
debugger/DebuggerThread.hx:258:
debugger/DebuggerThread.hx:259: 28
debugger/DebuggerThread.hx:260:
debugger/DebuggerThread.hx:261: Stopped?
debugger/DebuggerThread.hx:262:
```

Line 1398 in haxe.Serializer.cpp is `this->serialize(enumBase->_hx_getParamI(p));`

```
HXLINE( 482) {
HXLINE( 482) int _g5 = 0;
HXDLIN( 482) int _g6 = len;
HXDLIN( 482) while((_g5 < _g6)){
HXLINE( 482) _g5 = (_g5 + 1);
HXDLIN( 482) HX_VARI( int,p) = (_g5 - 1);
HXLINE( 483) this->serialize(enumBase->_hx_getParamI(p));
}
}
```

It looks like this line in the source code: https://github.com/HaxeFoundation/haxe/blob/6863504c3ad4a48e4fc2ef3e4a1ccb641658f0c6/std/haxe/Serializer.hx#L472

Contributor guide

Open the contributing guide

Research direction

Start at haxe.Serializer.run() and haxe.Serializer_obj::serialize(), particularly the generated C++ locations shown in the stack trace. Reproduce with a 31-element TEnum on the cpp target using the reported Windows build command, then verify serialization completes without a stack overflow and add coverage for the failing case if the repository's tests provide a suitable entry point.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.