Proposal: Reflect Type Transfer for the State Codec

Open
#2 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
25/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
go
Domain
backend

Research direction

Start with Save and Load in internal/state/state.go, then read encodeState.findType and decodeState.findType in internal/state/encode.go and internal/state/decode.go. Review writer.put and reader.get in internal/state/memory.go plus the existing nativeReflectType bridge. Done means the state codec discovers and transfers the described type table and restores object-graph type relationships without changing the listed out-of-scope components.

Written by the indexing model from the issue text.

Description

Proposal: Reflect Type Transfer for the State Codec

1. Summary

Discover types from Go's reflect caches, describe them using references to other types, and preload them into the state codec's own type table. Encode that table before the object graph in caller-owned shared memory. The guest reconstructs its type table first, then uses it to allocate and restore values, including values that themselves represent a reflect.Type.

There is one TypeID namespace per transfer. Cache discovery and object traversal use the same table. Type records are written in TypeID order; object traversal order does not determine how the guest interprets those IDs.

This is a design proposal, not a description of completed support. It applies to the imported gVisor state code in internal/state. It does not change Sentry, the shared-library bridge, or ixgo in this increment. Source details below were checked against this worktree and Go 1.26.6.

2. User Stories / Motivation

  • A caller passes an object containing a dynamically constructed type and values of that type. Both must refer to the same reconstructed type in the guest.
  • A caller passes several interface values with the same dynamic type. The guest must resolve their type consistently without manual state.Register calls.
  • Later ixgo integration needs ordinary guest reflect.Type objects that reflection can use directly. This proposal supplies type reconstruction; interpreter state and function reconstruction remain separate work.

The common need is to transport type identity and construction information alongside the existing object graph, without copying runtime type descriptors as ordinary heap objects.

The following example is used throughout the proposal:

recordType := reflect.StructOf([]reflect.StructField{
    {Name: "Count", Type: reflect.TypeOf(int(0)), Tag: `json:"count"`},
})
sliceType := reflect.SliceOf(recordType)

record := reflect.New(recordType).Elem()
record.Field(0).SetInt(42)
items := reflect.MakeSlice(sliceType, 1, 1)
items.Index(0).Set(record)

root := struct {
    Type  reflect.Type
    Value any
    Items any
}{
    Type:  recordType,
    Value: record.Interface(),
    Items: items.Interface(),
}

After restoration, these relationships must hold:

reflect.TypeOf(restored.Value) == restored.Type
reflect.TypeOf(restored.Items).Elem() == restored.Type
reflect.ValueOf(restored.Value).Field(0).Int() == 42
reflect.ValueOf(restored.Items).Index(0).Field(0).Int() == 42

The host and guest type descriptor addresses may differ. Their relationship is established through the transferred type table.

3. Current Workaround

The imported state implementation resolves registered types through StateTypeName, StateFields, and a global name-to-type database populated by Register. The native closure extension also has a nativeType path that looks up static types in executable DWARF, and a closureType path for capture layouts derived from a PC.

This supports existing registered types and some executable types, but does not describe arbitrary standard-reflect dynamic types. A StructOf result cannot acquire the StateTypeName and StateFields methods required by the existing registration API. Passing a type as a value also requires recognizing what that value represents, rather than recursively copying its private *reflect.rtype implementation.

The current entry points are Save and Load, encodeState.findType, and decodeState.findType. The current byte-slice transport is already writer.put and reader.get.

4. Goals

  • Enumerate the four standard reflect type caches and recursively collect the types referenced by their entries.
  • Use one state-owned TypeID table for discovered types, types encountered while encoding values, and restored types.
  • Reconstruct standard reflect dynamic types and preserve references to static types from the same executable.
  • Encode a reflect.Type value as a reference to that table.
  • Reuse the existing object graph machinery for values, shared references, interior references, and cycles.
  • Keep the shared-memory transport as w.put and r.get, without introducing JSON, gob, RPC, or a second serializer.

5. Out of Scope

  • Copying raw runtime type descriptors, runtime.reflectOffs, GC metadata, method code, or reflect's cache backing storage.
  • Discovering every type ever created through third-party unsafe runtime manipulation.
  • Reconstructing arbitrary dynamically named types, arbitrary dynamic interface types, or arbitrary method sets through standard reflect constructors.
  • Transferring channel contents, blocked goroutines, package globals, external resources, or interpreter execution state.
  • Implementing reflect.Value transfer or reflect.MakeFunc callback transfer. Their type references can use this table, but their values require separate handling.
  • Broadening the existing native closure ABI and platform support, changing Sentry, or connecting internal/state to production sandbox.Run as part of this document.

6. Proposal

6.1 Design Rule and Ownership

The codec owns the type table. The cache reader supplies reflect.Type objects to that table; it does not maintain a separate ID allocator. An object encoder requests the ID of a type from the same table. A decoder resolves that ID from the transferred records.

Host reflect caches ----> state type table <---- object traversal
                                  |
                         ordered type records
                                  |
                         caller-owned memory
                                  |
                         guest state type table
                                  |
                         restored object graph
Owner Responsibility
Reflect cache reader Read cache entries and expose their represented types; do not modify caches.
State type table Deduplicate types, assign IDs, describe dependencies, and resolve guest types.
State object codec Encode and decode values and object references using those IDs.
Shared-memory caller Own the buffer and keep it valid while Save or Load uses it.
Guest reflect/runtime Construct and retain valid local type descriptors and their runtime metadata.

The type table is local to one Save or Load operation. TypeID 7 in one transfer has no relationship to TypeID 7 in another. ObjectID is a separate namespace: it identifies an object, not its type.

6.2 Find Types in Reflect Caches

Go 1.26.6 stores standard reflect constructor results in four caches. Cache keys and hash buckets are lookup machinery; they are not transferable TypeIDs.

Symbol Actual stored values Extraction
reflect.ptrMap *abi.PtrType values in a sync.Map Obtain the pointer with reflect.ValueOf(value).UnsafePointer(); its initial abi.Type is at the same address; convert with the existing reflect.toType bridge.
reflect.lookupCache Values implementing reflect.Type in a sync.Map Assert value.(reflect.Type). This covers ArrayOf, ChanOf, MapOf, and SliceOf.
reflect.funcLookupCache.m Buckets of []*abi.Type Iterate each bucket through reflection and convert each pointer with reflect.toType.
reflect.structLookupCache.m Buckets of []reflect.Type Iterate each bucket directly.

The function and struct caches have an outer sync.Mutex followed by a sync.Map. A linkname declaration must mirror that layout, including the mutex. For example, the already explored discovery shape is:

// These declarations depend on the checked Go version's private layout.
// The declaring file imports reflect, sync, and unsafe.

//go:linkname pointerCache reflect.ptrMap
var pointerCache sync.Map

//go:linkname compositeCache reflect.lookupCache
var compositeCache sync.Map

//go:linkname functionCache reflect.funcLookupCache
var functionCache struct {
    sync.Mutex
    Entries sync.Map
}

//go:linkname structCache reflect.structLookupCache
var structCache struct {
    sync.Mutex
    Entries sync.Map
}

Access to these private cache variables requires -ldflags=-checklinkname=0 with the checked toolchain. This permits linking; it does not establish layout compatibility with another Go version.

Reuse nativeReflectType, the existing linkname declaration for reflect.toType; another conversion bridge is unnecessary. Enumeration uses Range on each map and iterates every element of collision buckets. For example:

pointerCache.Range(func(_, value any) bool {
    collect(nativeReflectType(reflect.ValueOf(value).UnsafePointer()))
    return true
})

compositeCache.Range(func(_, value any) bool {
    collect(value.(reflect.Type))
    return true
})

functionCache.Entries.Range(func(_, value any) bool {
    bucket := reflect.ValueOf(value)
    for i := 0; i < bucket.Len(); i++ {
        collect(nativeReflectType(bucket.Index(i).UnsafePointer()))
    }
    return true
})

structCache.Entries.Range(func(_, value any) bool {
    for _, typ := range value.([]reflect.Type) {
        collect(typ)
    }
    return true
})

Here collect means append the type if it is not already in a map[reflect.Type]bool. It does not mean assign an independent cache ID. Collect actual reflect.Type references, not only uintptr addresses, so the discovery result holds valid Go references until encoding finishes.

Complete the cache reads before processing the collected roots. Do not call type constructors from the Range callbacks: some constructors create more cached types, including internal map layout types.

The enumeration reads all entries it observes. It is not a single atomic snapshot across caches. Go's function and struct cache buckets are append-only in this version, but sync.Map.Range does not guarantee a consistent snapshot under concurrent writes. Object traversal must still add referenced types absent from the discovery result. Strict enumeration of every cache entry at one instant requires quiescent type creation; the synchronization policy is an open question in section 12.

These caches are not a complete registry of all static types. For example, PointerTo can return a compiler-provided PtrToThis before touching ptrMap. They also do not contain all types created by third-party unsafe constructors. Static dependencies and value-reachable types are collected through the same type table as encoding proceeds. runtime.reflectOffs is not a substitute: it is a separate mapping that also contains names and other offsets.

6.3 Assign IDs Once, Inside State

Keep the existing concept of a state-owned map[reflect.Type].... Equality of reflect.Type identifies a type; Type.String() is not a unique identity key.

The logical algorithm is:

reserve(t):
    if the state table already contains t:
        return its existing ID
    assign the next nonzero ID
    store t -> ID immediately
    reserve the record slot for that ID
    queue t for description
    return ID

preload:
    for each deduplicated cache type t:
        reserve(t)

describe queued types:
    for each type t:
        construct its record
        for each referenced type child:
            use reserve(child) as the reference
        put the completed record in t's reserved slot

reserve is pseudocode for the table operation, not a proposed exported API. Reserve before describing children, so repeated references reuse an ID. Newly discovered types during object encoding go through the same operation. Finish describing queued types before writing the stream.

For the running example, suppose the relevant discovered roots arrive as [sliceType, recordType]. In a minimal table with these roots, the result is:

Preload roots:          sliceType -> 1, recordType -> 2
Describe ID 1:          Slice { Elem: 2 }
Describe ID 2:          Struct { Count: reserve(int), Tag: json:"count" }
Add dependency:         int -> 3
Describe ID 3:          Builtin { Kind: Int }

Real cache enumeration can contain unrelated entries, so the numeric IDs may differ. Correctness depends on the recorded references, not these particular numbers or deterministic cache iteration.

Now state reaches root.Value before root.Items:

root.Value has recordType -> lookup returns 2
root.Items has sliceType  -> lookup returns 1

State does not renumber recordType to 1 just because that value was visited first. Both the cache reader and the value encoder use the same table.

6.4 Decompose Each Type

The input is a reflect.Type t whose ID has already been reserved. The output is its semantic type record in that ID's slot. Every reserve(child) below uses the same state table from section 6.3; it returns the child's ID and queues any newly discovered type for description. Guest notation types[id] means the result of resolving that dependency, not a requirement that records arrive in dependency order.

Apply the identity checks before switching on Kind: recognize an actual builtin, otherwise preserve an executable static type, otherwise reject an unsupported non-static named type, then describe a supported dynamic type. This prevents type Count int from becoming int or a named struct from becoming an anonymous struct.

Input type / classification Exact host extraction Semantic record stored in this TypeID slot Guest reconstruction Example / boundary
Nil type value Check t == nil before asking for Kind or reserving a type No type record; the value codec preserves nil Restore the nil value ID 0 is not assigned to a real type. Invalid is not a valid type descriptor to reconstruct.
Predeclared bool Confirm type identity with the builtin bool type Builtin { Kind: Bool } Retrieve guest builtin bool No child TypeID.
Predeclared signed integers Match the builtin int, int8, int16, int32, or int64 type by identity Builtin { Kind: t.Kind() } Retrieve that exact guest builtin Preserve Int versus Int64, even on a 64-bit ABI.
Predeclared unsigned integers Match the builtin uint, uint8, uint16, uint32, uint64, or uintptr type by identity Builtin { Kind: t.Kind() } Retrieve that exact guest builtin byte is an alias for uint8, not another type entry.
Predeclared floats and complex numbers Match float32, float64, complex64, or complex128 by identity Builtin { Kind: t.Kind() } Retrieve that exact guest builtin The Kind distinguishes each precision.
Predeclared string Confirm type identity with the builtin string type Builtin { Kind: String } Retrieve guest builtin string String contents belong to value encoding.
unsafe.Pointer Confirm identity with the unsafe.Pointer type Builtin { Kind: UnsafePointer } Retrieve guest unsafe.Pointer type Reconstructing this type does not make arbitrary pointer values transferable.
Any remaining type in the current executable's static type region Obtain the descriptor address and subtract host runtime.types Static { Offset: descriptorAddress - typeRegionBase } Resolve the validated descriptor at the corresponding guest offset Includes named types, static interfaces such as error, recursive named types, and unnamed static types; their methods remain part of the original descriptor.
Non-static named type After builtin/static checks, test t.Name() != "" Unsupported-type error; no anonymous replacement No standard reflect constructor for an arbitrary named type Name and PkgPath alone cannot recreate its identity or attach its methods.
Dynamic pointer *T elemID = reserve(t.Elem()) Pointer { Elem: elemID } reflect.PointerTo(types[elemID]) The type record contains no pointee object address.
Dynamic slice []T elemID = reserve(t.Elem()) Slice { Elem: elemID } reflect.SliceOf(types[elemID]) Length, capacity, and backing storage describe a slice value, not its type.
Dynamic array [N]T length = t.Len(); elemID = reserve(t.Elem()) Array { Len: length, Elem: elemID } reflect.ArrayOf(length, types[elemID]) [3]int and [4]int have different type records.
Dynamic map map[K]V keyID = reserve(t.Key()); elemID = reserve(t.Elem()) Map { Key: keyID, Elem: elemID } reflect.MapOf(types[keyID], types[elemID]) Key comparability follows from the restored key type; guest MapOf builds its own hasher and internal layout.
Dynamic channel direction = t.ChanDir(); elemID = reserve(t.Elem()) Chan { Dir: direction, Elem: elemID } reflect.ChanOf(direction, types[elemID]) Preserve BothDir, SendDir, or RecvDir. Buffer capacity and queued values are not type metadata.
Dynamic function signature For i = 0..NumIn()-1, append reserve(t.In(i)); for i = 0..NumOut()-1, append reserve(t.Out(i)); read t.IsVariadic() Func { In: inputIDs, Out: outputIDs, Variadic: flag } reflect.FuncOf(resolvedInputs, resolvedOutputs, flag) For func(string, ...int) error, In refers to string and []int; Out refers to static error; Variadic is true. No PC or captured environment is stored here.
Dynamic anonymous struct For i = 0..NumField()-1, get f = t.Field(i) and preserve f.Name, f.PkgPath, reserve(f.Type), string(f.Tag), and f.Anonymous in that order Struct { Fields: [{ Name, PkgPath, TypeID, Tag, Anonymous }, ...] } Resolve each TypeID, build the ordered []reflect.StructField, then call reflect.StructOf(fields) Preserve the complete tag, package path, field order, and embedding flag; obey StructOf's field and method restrictions.
Non-static interface Kind is Interface after the static-type check Unsupported-type error There is no standard reflect.InterfaceOf constructor Method names and signatures are inspectable, but that does not provide a way to construct the interface.
Other non-static type outside these supported cases No matching identity or supported constructor description Unsupported-type error No implicit conversion to an approximate type Covers types requiring third-party runtime-specific reconstruction.

The record names and field labels above describe the format's meaning; they do not assign binary tag numbers. The complete decomposition algorithm is:

describe(t):
    require t != nil and t already has an ID

    if t equals one of the actual builtin types:
        return Builtin { Kind: t.Kind() }

    if t's descriptor belongs to the current executable's static type region:
        return Static { Offset: descriptorAddress(t) - typeRegionBase }

    if t.Name() != "":
        fail unsupported non-static named type

    switch t.Kind():
        Pointer:
            return Pointer { Elem: reserve(t.Elem()) }
        Slice:
            return Slice { Elem: reserve(t.Elem()) }
        Array:
            return Array { Len: t.Len(), Elem: reserve(t.Elem()) }
        Map:
            return Map { Key: reserve(t.Key()), Elem: reserve(t.Elem()) }
        Chan:
            return Chan { Dir: t.ChanDir(), Elem: reserve(t.Elem()) }
        Func:
            inputs = []
            outputs = []
            for i from 0 up to, but not including, t.NumIn():
                inputs.append(reserve(t.In(i)))
            for i from 0 up to, but not including, t.NumOut():
                outputs.append(reserve(t.Out(i)))
            return Func { In: inputs, Out: outputs, Variadic: t.IsVariadic() }
        Struct:
            fields = []
            for i from 0 up to, but not including, t.NumField():
                f = t.Field(i)
                fields.append({
                    Name:      f.Name,
                    PkgPath:   f.PkgPath,
                    TypeID:    reserve(f.Type),
                    Tag:       string(f.Tag),
                    Anonymous: f.Anonymous,
                })
            return Struct { Fields: fields }
        Interface:
            fail unsupported non-static interface type
        otherwise:
            fail unsupported type

while the description queue is not empty:
    t = take next queued type
    id = stateTypeIDs[t]
    records[id - 1] = describe(t)
    # reserve(child) may have appended more types to this same queue.

For the running example, describing recordType at ID 2 produces the following complete field record:

t.Field(0) gives:
  Name="Count", PkgPath="", Type=int, Tag=`json:"count"`, Anonymous=false

reserve(int) -> 3

records[1] = Struct {             # records[1] is the slot for TypeID 2
    Fields: [{
        Name: "Count",
        PkgPath: "",
        TypeID: 3,
        Tag: `json:"count"`,
        Anonymous: false,
    }],
}

Offset and Index from a StructField are not encoded construction inputs; guest StructOf computes them. Size, alignment, pointer metadata, and equality machinery are also derived locally. Do not copy host addresses for type names, Equal/Hasher callbacks, GC bitmaps, or reflectOffs entries.

The static branch preserves a compiled type Node struct { Next *Node } as one reference instead of recursively rebuilding its named definition. As specified in section 12, validating the guest's static descriptor target remains unresolved; the pseudocode's host classification does not authorize casting an arbitrary guest offset to a runtime type pointer.

6.5 Encode Type Values and Ordinary Values

Use the same type table for two different references:

Interface { DynamicType: 2, Value: ... }
    means: this ordinary value has type 2

TypeValue { ReferencedType: 2 }
    means: this value represents type 2

The distinction is necessary for root.Type versus root.Value. A value implementing reflect.Type must be recognized before ordinary traversal follows the implementation pointer into runtime metadata. This also applies when that type value is carried through an any interface or used as a map key.

For the example, the logical object data is:

root.Type  = TypeValue(2)
root.Value = Interface(DynamicType=2, Value=StructFields[42])
root.Items = Interface(DynamicType=1, Value=Slice(length=1, capacity=1, ...))
slice backing data = [StructFields[42]]

These are semantic diagrams. The existing object codec continues to emit ObjectIDs and references for the root, slice backing storage, and other objects. The diagram does not introduce field names into every object record or replace state pointer tracking.

After a dynamic interface type is resolved, ordinary fields still use the existing reflection traversal. Static field types generally follow from the parent object's type, so a separate TypeID need not be added to every scalar field value.

Native closure values continue through the existing PC and captured-environment path. Any types needed for their values must resolve consistently through the type table. Synthetic capture layouts derived from DWARF retain their existing special handling; this proposal does not establish new capture layouts or make callback migration automatic.

6.6 Write the Shared-Memory Stream

The current state encoder builds its object graph before writing it. Keep that sequencing:

1. Collect cache roots.
2. Preload state's type table and describe their dependencies.
3. Traverse the root object graph, reusing or adding type IDs.
4. Finish pending type descriptions and freeze the table for this write.
5. Write the existing object-count header.
6. Write type records in ascending TypeID order through w.put.
7. Write object IDs and their encoded values through w.put.

The type section uses the existing position-based ID convention:

shared memory
  object-count header
  first type record   = Slice(Elem=2)                         -> ID 1
  second type record  = Struct(Count: TypeID=3, json:"count")   -> ID 2
  third type record   = Builtin(Int)                          -> ID 3
  remaining type records for the complete graph, if any
  object ID, object data
  object ID, object data
  ...

There is no separate cache-number table and no redundant explicit ID field required in each type record. The first record is ID 1, the second is ID 2, and so on. A description that recursively finishes before its parent must still be written in its reserved ID slot. Appending records in description-completion order would corrupt references.

Type records and object IDs already have distinct record kinds. Keep the existing object-count header: it counts objects, not type records. The guest can collect the leading type records until it reads the first object ID, retaining that first ID for subsequent object decoding. Numeric record tags and the exact new payload encoding remain implementation choices to settle before code changes; this proposal specifies their meaning and ordering.

The writer fills the caller-owned byte slice. Buffer exhaustion returns the existing short-buffer error; it does not allocate a replacement mapping. Descriptors are encoded data in shared memory, not live runtime type objects placed at agreed virtual addresses.

6.7 Decode Descriptions, Then Reconstruct Types

The guest must not immediately construct each type as its record arrives. Forward references are valid: record 1 above depends on record 2, which depends on record 3.

read record 1 -> descriptions[1] = Slice(Elem=2)
read record 2 -> descriptions[2] = Struct(Count: TypeID=3, ...)
read record 3 -> descriptions[3] = Builtin(Int)
read remaining type records
read first object ID -> the type prelude has ended

Only then resolve the type graph. A per-ID status distinguishes not started, resolving, and complete. A completed entry returns its stored reflect.Type; a dependency outside the table is an error. A cycle requiring construction of an unfinished dynamic type is unsupported by the standard constructors and must not recurse indefinitely.

For the example:

resolve(1)
  needs resolve(2)
    needs resolve(3)
      types[3] = guest int
    types[2] = StructOf([{Name:"Count", Type:types[3], Tag:`json:"count"`}])
  types[1] = SliceOf(types[2])

For static records, compute the candidate from the guest's executable type-region base and the encoded offset, then resolve it through validated static-type handling. Matching executable and ABI are required; an arbitrary in-range address is not proof that it is a valid type descriptor. The current DWARF index is not exhaustive, so completing that validation path is an explicit open question rather than an assumed unsafe cast.

Standard reflect constructors perform their own runtime integration and cache insertion. There is no additional public reflect Register step, and host cache IDs or reflectOffs IDs are never installed into the guest runtime.

6.8 Decode Values Using the Reconstructed Table

Once the table is ready, continue with the saved first object ID and the existing object decoding loop:

root.Type:
  TypeValue(2)
  -> obtain types[2]
  -> set root.Type to that reflect.Type value

root.Value:
  Interface(DynamicType=2, ...)
  -> reflect.New(types[2]).Elem()
  -> decode Count = 42
  -> set root.Value to the resulting value

root.Items:
  Interface(DynamicType=1, ...)
  -> restore the slice and backing objects using the existing graph decoder
  -> element type is types[1].Elem(), which is types[2]

Repeated references to ID 2 always resolve to the same entry. Independently, repeated ObjectIDs preserve shared objects. Equal types do not imply identical objects, and two separate values of the same type must not be merged.

The guest reconstructs types and values in its own Go runtime. The shared buffer contains their descriptions and data. It is not installed as the runtime type heap. The per-load table keeps references while decoding, and the resulting values and runtime caches retain the types they use.

6.9 Concrete Integration Points in State
File / existing boundary Required change
state.go: Save and Load setup Create the per-transfer table, preload discovered types before object traversal, and prepare guest types before decoding values. Preserve caller-owned buffer behavior.
types.go: encode/decode type databases Use one ID allocator and ordered descriptor slots; resolve guest types from records instead of requiring the global name registry.
encode.go: findType and pending types Return the preassigned ID when present; append newly encountered types through the same allocator; emit descriptors by ID.
encode.go: value dispatch Recognize values representing reflect.Type before traversing their private implementation.
decode.go: Load, findType, interface decoding Read the full type prelude, resolve dependencies, then reuse the resulting reflect.Type objects when allocating values.
object.go: type and object records Encode richer type descriptions and the distinct type-value reference through the existing object serialization machinery.
native.go: executable type lookup Supply static-type resolution without conflating type reconstruction with native closure capture discovery.
memory.go Retain the existing byte-slice writer and reader; no alternate transport is needed.

The cache reader is a Go-version-specific part of this type-handling boundary, not a new public subsystem. No second copy of object graph traversal is introduced.

Removing manual Register as a type-discovery requirement does not automatically remove the semantics of StateSave, StateLoad, field reconciliation, or AfterLoad callbacks. They may describe logical state differently from Go struct fields. Their retention and migration policy must be decided explicitly; they must not be silently bypassed as a side effect of switching type tables.

6.10 Boundary Cases and Concurrency
Case Meaning for this design
Cache order differs between runs IDs can differ; each stream carries its own ordered type definitions.
Guest cache already contains the type Constructors may return the existing canonical type; install that result under the stream ID.
A value references a type missed by cache enumeration Add it through the same state type table during graph traversal.
A static named type has the same Kind as a builtin Preserve the static type identity; do not replace it with the builtin.
A reflect.Type appears in an interface or map key Encode the represented type reference, not *reflect.rtype storage.
A type record describes a channel or function Reconstruct the type only; existing limitations on channel and function values still apply.
StructOf encounters fields or promoted methods it does not support Preserve the constructor boundary and report failure; do not silently drop fields or methods.
Another goroutine creates types while caches are enumerated Reads are concurrent map reads, not an atomic inventory; reachable types are supplemented during encoding.
Another goroutine mutates the values being saved This proposal supplies no atomic value snapshot or new STW mechanism; the caller must provide a stable graph for Save.
Multiple transfers run concurrently Their tables and IDs are independent. They read global caches and may populate guest reflect caches through normal constructors.

Scanning all observed cache types means work and stream size grow with the process's accumulated cache contents, including types unrelated to the current root. Guest reconstruction can also retain those types in reflect's global caches. This cost follows from the selected full-cache discovery approach; a future reachable-only optimization must not be assumed here.

7. Error Handling

Condition Behavior
Unknown or out-of-range TypeID Return a decoding error before using it to construct or allocate a value.
Truncated stream or insufficient output buffer Preserve the reader/writer error through the existing state error boundary.
Type records appear after object decoding has begun Reject the stream; all type definitions belong to the prelude.
Unsupported dynamic named/interface type or construction cycle Return an unsupported-type error identifying the relevant record or type.
Reflect constructor rejects its arguments Return a type-reconstruction error; do not substitute a different type.
Executable, Go ABI, or static-type reference cannot be validated Stop before constructing values with that type.
A value is incompatible with its referenced type Return a decoding error; do not force an unsafe assignment.

Concrete diagnostics and binary tag values are not frozen by this proposal. Validation of shared-memory lengths, type references, and static descriptors belongs at the decoding boundary. Checking an ELF address range alone does not validate a runtime type object.

8. Compatibility

The semantic type record format changes, so encoder and decoder must be updated together. This proposal does not introduce a cross-version stream compatibility layer. The exact format-version treatment is listed as an open question before implementation.

The repository currently targets Go 1.26.6. Its native closure path requires the same non-PIE executable, capture DWARF, and Linux amd64 or arm64. Cache enumeration was investigated separately from that path; its use of private Go symbols is also version dependent. This proposal does not turn a cache enumeration result into proof of full closure transfer support on another platform.

No production sandbox or Sentry API change is required to write the type table into the existing shared-memory buffer. Connecting this refactored internal codec to the production transfer path remains a separate integration step.

9. Alternatives Considered

9.1 Keep Manual State Registration for Every Type

The registration contract requires methods on the registered type. Runtime-created anonymous types cannot supply those methods, so caller-side registration alone cannot implement this feature.

9.2 Allocate Cache IDs Separately from State IDs

This creates two meanings for an ID and requires translation on every type reference. Preloading state's own table removes that duplication.

9.3 Copy Runtime Type Memory or reflectOffs

Runtime descriptors reference other descriptors, names, code, and runtime-managed metadata. reflectOffs is neither a contiguous type region nor a complete registry. Reconstructing standard dynamic types through their constructors gives the guest valid local metadata.

9.4 Number Types Again in Guest Cache Order

Cache iteration order is not stable or shared between processes. The stream's ordered type records define the IDs; guest cache order has no role in decoding.

9.5 Serialize Only Type Names

Type.String is not a unique identity key, and a name does not contain the construction inputs for a dynamic composite type. Named static types also need their original identity and methods, not an anonymous approximation.

10. Testing Strategy

These are acceptance tests for implementation, not checks claimed to have passed for this proposal.

Test Required assertion
Enumerate all four cache shapes Find constructor-created pointer, slice, array, map, channel, function, and struct types; visit every hash-bucket entry.
Cache-first numbering, reverse value order Preload B then A, encode A then B, and restore the right types without renumbering.
Forward type references Reconstruct Slice #1 -> Struct #2 -> Int #3 after reading all descriptions.
Repeated type identities All uses of a type share one TypeID; restored Type values equal the dynamic types of restored values.
Builtins versus named types Preserve a static type Count int as distinct from int; preserve aliases according to Go type identity.
Seven dynamic constructor categories Check element/key relationships, array length, channel direction, function parameter order and variadic flag, and struct field metadata.
Struct layout On the supported ABI, verify reconstructed Size, Align, field Offset, tags, and embedded-field status against the source type.
Static interfaces and recursive named types Preserve the original type identity and method set; do not reconstruct them as anonymous types.
Type values in interfaces, maps, and repeated fields Round-trip represented types without walking runtime descriptors.
Object graph regressions Existing alias, cycle, interior-reference, and native-closure tests continue to preserve object relationships.
Concurrent cache growth No unsafe cache access; missed but value-reachable types are added before the stream is written. Do not assert an atomic cache snapshot.
Consecutive and concurrent transfers No cross-transfer TypeID leakage or dependency on prior per-transfer tables.
Guest reconstruction and GC Restored values remain valid after the decode table is released and GC runs.
Malformed records and short buffers Fail through the state error boundary, including invalid IDs, invalid constructors, unsupported cycles, and invalid static references.
Existing custom state hooks Verify the separately agreed retention/migration policy before replacing their lookup path.
Linux amd64 and arm64 with the checked Go version Validate the codec path on both existing native-closure platforms; do not infer either result from cache enumeration alone.

11. Summary of Changes

Area Proposed result
Discovery Read standard reflect caches and supplement their entries with referenced and encountered types.
Type identity One state-owned, per-transfer TypeID table, preloaded before ordinary object traversal.
Shared memory Ordered type descriptions followed by existing object records, written with w.put.
Guest reconstruction Read descriptions first, resolve type dependencies, then decode values.
reflect.Type values Encode references to represented types instead of copying runtime descriptor objects.
Register Remove it as a prerequisite for type lookup; explicitly decide the remaining custom state semantics before deleting related code.
Sentry and ixgo No changes in this increment; later adapters can consume restored guest reflect.Type objects.

12. Open Questions Before Implementation

  1. Static descriptor validation. The existing native metadata indexes types found in DWARF, which is not a complete inventory of static types. The static-offset scheme needs a concrete way to accept valid executable types missing from that index while rejecting offsets that are not type descriptors. This proposal does not approve an unchecked cast.
  2. Custom state hooks. Decide whether StateSave/StateLoad and logical field reconciliation remain supported, and how their metadata shares the single type table. Type reconstruction alone is not authorization to remove those behaviors.
  3. Strict cache snapshot semantics. Decide whether the requirement is all entries observed during enumeration plus all value-reachable types, or an atomic inventory of all caches. The latter needs coordination of type creation; repeated Range calls do not prove completeness.
  4. Binary format details. Choose concrete record tags and the incompatible-format handling before editing the codec. Preserve the agreed implicit TypeID ordering; do not add a second numbering scheme.
  5. Types outside the standard constructors. Classify any ixgo/xtype-created named types before claiming this path covers them. Reconstructing a standard anonymous type does not establish equivalence to a third-party named type.

13. Source References

Repository references are relative to this proposal. Standard-library paths refer to the checked Go 1.26.6 source tree, not an assumed stable private API.

Source Evidence used
internal/state/types.go Current ID allocation, positional descriptor registration, global name registry, and custom type reconciliation.
internal/state/encode.go Type lookup, interface encoding, graph traversal before writing, and type records preceding objects.
internal/state/decode.go Descriptor consumption, type resolution, interface allocation, and deferred object reconstruction.
internal/state/object.go Current type descriptions, type specifications, object tags, and binary serialization.
internal/state/memory.go Caller-owned buffer, w.put/r.get, short-buffer and truncated-input behavior.
internal/state/native.go Existing reflect.toType bridge, executable metadata, PC/capture handling, and ABI restrictions.
src/reflect/type.go: ptrMap, lookupCache, funcLookupCache, structLookupCache Cache ownership, entry shapes, constructor metadata, field information, and cache insertion.
src/reflect/map.go: MapOf, groupAndSlotOf Actual map constructor, local hasher construction, and additional cached internal layout types.
src/reflect/makefunc.go: makeFuncImpl, MakeFunc Function signature metadata is separate from callback implementation and environment.
src/runtime/type.go: reflectOffs Offset mappings are not a complete dynamic type registry.
src/sync/map.go: Range Enumeration does not provide a consistent snapshot under concurrent updates.
Dominant language
Go
Stars
0
Forks
1
Avg merge
6h 36m
Merged PRs (30d)
18

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from xgo-dev/sandbox

All issues in xgo-dev/sandbox

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.