BattletechModders / BattletechModders/ModTek
Improve json parsing performance
- Dominant language
- C#
- Stars
- 137
- Forks
- 40
- Avg merge
- 14h 41m
- Merged PRs (30d)
- 1
Description
TLDR: Too much effort, therefore postponed
currently BT loads jsons
1. BT request resources via datamanager
2. datamanager loads json as string
3. target class converts string to nested dictionary
4. helper utility then maps dictionary to target object via reflection
- there is some lookup-caching but its still meta-code working via reflection apis
an improvement could look like
1. BT requests resources
2. maps a json file (bytes) to target object
- use cached IL-processing/mapping code (see msgpack which partially does that)
- removes costly meta-processing (if/else branches etc..)
- parse bytes with help of sse or avx512 simd
- simdjon is a very fast parser. Issues:
- C# <-> native overhead is large
- the .NET only solution is not for .NET Framework, would require backport
- span etc.. is already backported, no issue
- missing SIMD support in the mono jit employed by Unity
- is only the parser, not the mapper
- fix: probably have to implement ourselves just for BT' .NET version
- never parsing to a string unless a string is actually needed
- json is utf8, while .NET strings are utf16, conversion always costs
- enums etc.. could be cached to their utf8 byte[] representation
- the fastJson parser used in vanilla code and custom mapper by HBS have some deterministic but undefined behavior regarding order of field/property initialization. One would need to replicate the same order as vanilla to avoid NREs or other logic errors.
other ideas:
- use newtonsoft json not only for merging but also datamanager loading
- newton is much (3-6x) slower than the fastJson+custom mapper solution
- newton is also not fully compatible, order still has to be fixed but otherwise this branch has much solved already (custom converters, settings etc..): https://github.com/BattletechModders/ModTek/tree/working-on-newtonsoft-compatibility
- msgpack would still be fastest, but that would only work for a cache, as modders cant use notepad to modify data
- see cmission/kmission's CustomPrewarm as an example on how to cache
- much is hardcoded, could be made generic with some effort in msgpack setup for resolver+formatters (and CC+ME being msgpack compatible); could then be integrated into modtek
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.