Improve struct dictionary codecs in Go
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 11
- Forks
- 6
- Avg merge
- 55m
- Merged PRs (30d)
- 1
Description
We do a lot of unnecessary copying and cloning of dictionary-encoded structs currently, especially during encoding.
Consider the following:
- Allow marking structs frozen. Once frozen they can be safely referenced by pointer from multiple places (including dictionaries) without cloning.
- Use freezing after record is read.
- Allowing referencing frozen structs from read operations directly in write operations. Should improve copying (read/write).
- Make CopyFrom smarter, avoid cloning frozen structs, just copy the pointer.
- Encourage Freeze() when creating data (not from Read()) of Writer. Should improve encoding.
- Cache refNum in the object to avoid slow Tree lookups.
- Replace Tree by Hashmap. Cache the hashcode, invalidate on modifications.
- Allowing setting dictionary structs in their containing structs by pointer instead of requiring full CopyFrom or field-by-field setting.
Most of these can be applied to varying degrees both to Go and Java. This summary issue is for Go. Duplicate it for Java.
See some initial experiment on this branch https://github.com/splunk/stef/compare/main...tigran/profileserialize?expand=1
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the Go struct and dictionary codec implementation and the initial experiment in the linked profileserialize branch. Work through the unchecked Tree-to-hashmap item alongside the completed freezing, pointer-reference, CopyFrom, and refNum goals. Done means the remaining dictionary-encoded struct copying is reduced without breaking the listed behaviors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- performance
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100