BattletechModders / BattletechModders/ModTek

Improve json parsing performance

Open
#239 0 comments 0 reactions 0 assignees View on GitHub
performance
Dominant language
C#
Stars
137
Forks
40
Avg merge
14h 41m
Merged PRs (30d)
1

Description

TLDR: Too much effort, therefore postponed

currently BT loads jsons
1. BT request resources via datamanager
2. datamanager loads json as string
3. target class converts string to nested dictionary
4. helper utility then maps dictionary to target object via reflection
- there is some lookup-caching but its still meta-code working via reflection apis

an improvement could look like
1. BT requests resources
2. maps a json file (bytes) to target object
- use cached IL-processing/mapping code (see msgpack which partially does that)
- removes costly meta-processing (if/else branches etc..)
- parse bytes with help of sse or avx512 simd
- simdjon is a very fast parser. Issues:
- C# <-> native overhead is large
- the .NET only solution is not for .NET Framework, would require backport
- span etc.. is already backported, no issue
- missing SIMD support in the mono jit employed by Unity
- is only the parser, not the mapper
- fix: probably have to implement ourselves just for BT' .NET version
- never parsing to a string unless a string is actually needed
- json is utf8, while .NET strings are utf16, conversion always costs
- enums etc.. could be cached to their utf8 byte[] representation
- the fastJson parser used in vanilla code and custom mapper by HBS have some deterministic but undefined behavior regarding order of field/property initialization. One would need to replicate the same order as vanilla to avoid NREs or other logic errors.

other ideas:
- use newtonsoft json not only for merging but also datamanager loading
- newton is much (3-6x) slower than the fastJson+custom mapper solution
- newton is also not fully compatible, order still has to be fixed but otherwise this branch has much solved already (custom converters, settings etc..): https://github.com/BattletechModders/ModTek/tree/working-on-newtonsoft-compatibility
- msgpack would still be fastest, but that would only work for a cache, as modders cant use notepad to modify data
- see cmission/kmission's CustomPrewarm as an example on how to cache
- much is hardcoded, could be made generic with some effort in msgpack setup for resolver+formatters (and CC+ME being msgpack compatible); could then be integrated into modtek

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.