apache / apache/fory

[Lua] Add Fory Serialization Support for Lua

Open
#3,380 3 comments 0 reactions 0 assignees View on GitHub
enhancement lua
Dominant language
Java
Stars
4.5k
Forks
443
Avg merge
5h 59m
Merged PRs (30d)
77

Description

### Feature Request

Add Apache Fory xlang serialization support for Lua.

### Is your feature request related to a problem? Please describe

_No response_

### Describe the solution you'd like

#### Design Plan

Implement a Lua xlang runtime that is wire-compatible with the xlang specification and implementation guide.

#### Scope and Deliverables

- Xlang encoder/decoder with little-endian binary format.
- Type registry for numeric and named user types.
- Struct/enum/union serialization support.
- Schema-consistent mode and compatible mode (meta share / TypeDef).
- Cross-language compatibility tests against existing implementations.
- Metatable restoration during deserialization for Lua struct-like objects.

#### Protocol Compatibility Requirements

- Header bitmap flags: `null`, `xlang`, `oob`.
- Reference flags: `NULL(-3)`, `REF(-2)`, `NOT_NULL(-1)`, `REF_VALUE(0)`.
- Internal type IDs (`0~56`) and separate `user_type_id` varuint32 for registered user types.
- Type meta rules for `NAMED_*`, `COMPATIBLE_STRUCT`, `TYPED_UNION`, and shared TypeDef streaming.
- Meta string encoding + dedup.
- Value encoding for varints, strings, collections/maps, struct field order, and union payload.

#### Lua-Specific Design Points

- Baseline runtime: Lua 5.4 (Lua 5.3 compatible where possible).
- Optional fast path: LuaJIT/FFI acceleration, pure Lua remains the compatibility baseline.
- Numeric policy: explicit int64/uint64 handling via wrappers where Lua number precision is insufficient.
- Table disambiguation: declared type descriptors for list/map/struct resolution.
- Reference tracking: identity tracking for reference-like values (`table`).

#### Metatable Restoration Strategy

- Register struct-like descriptors with `namespace/type_name` (or numeric type ID), `create_instance(ctx)`, and metatable metadata.
- Decode flow for struct-like values:
1. Resolve descriptor from type meta.
2. Allocate instance via `create_instance`.
3. Set metatable immediately.
4. Reserve/fill reference slot around field decode for cycle-safe graphs.
5. Populate fields using schema-consistent or compatible-mode mapping.
- For unknown named types in compatible mode, fallback to plain table while preserving decoded fields.

#### Implementation Phases

- Phase 0: bootstrap API and error taxonomy.
- Phase 1: buffer, endian IO, varints, header.
- Phase 2: reference resolver + type ID/meta core.
- Phase 3: primitive/string/time/binary serializers.
- Phase 4: list/set/map/array serializers.
- Phase 5: meta string + TypeDef.
- Phase 6: enum/struct/union.
- Phase 7: skip logic + malformed-input hardening.
- Phase 8: performance optimization and benchmarks.

#### Testing Plan

- Lua unit tests for buffer/varint/string/ref/type-meta.
- Comprehensive serialization tests for lua.
- Cross-language tests:
- Java <-> Lua.
- See java/fory-core/src/test/java/org/apache/fory/xlang/XlangTestBase.java.

#### Milestone Exit Criteria

- M1: buffers/primitives/strings/header/varints.
- M2: collections/maps/reference tracking complete.
- M3: struct/enum/union complete (schema-consistent mode).
- M4: compatible mode + TypeDef meta-share complete.
- M5: cross-language CI green with baseline perf report.

### Describe alternatives you've considered

_No response_

### Additional context

Reference implementation plan document in repository:

- https://fory.apache.org/docs/next/specification/xlang_serialization_spec
- https://fory.apache.org/docs/next/specification/xlang_implementation_guide

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.