splunk / splunk/stef

Improve struct dictionary codecs in Go

Open
#153 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Go
Dominant language
Java
Stars
11
Forks
6
Avg merge
55m
Merged PRs (30d)
1

Description

We do a lot of unnecessary copying and cloning of dictionary-encoded structs currently, especially during encoding.

Consider the following:

  • Allow marking structs frozen. Once frozen they can be safely referenced by pointer from multiple places (including dictionaries) without cloning.
  • Use freezing after record is read.
  • Allowing referencing frozen structs from read operations directly in write operations. Should improve copying (read/write).
  • Make CopyFrom smarter, avoid cloning frozen structs, just copy the pointer.
  • Encourage Freeze() when creating data (not from Read()) of Writer. Should improve encoding.
  • Cache refNum in the object to avoid slow Tree lookups.
  • Replace Tree by Hashmap. Cache the hashcode, invalidate on modifications.
  • Allowing setting dictionary structs in their containing structs by pointer instead of requiring full CopyFrom or field-by-field setting.

Most of these can be applied to varying degrees both to Go and Java. This summary issue is for Go. Duplicate it for Java.

See some initial experiment on this branch https://github.com/splunk/stef/compare/main...tigran/profileserialize?expand=1

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the Go struct and dictionary codec implementation and the initial experiment in the linked profileserialize branch. Work through the unchecked Tree-to-hashmap item alongside the completed freezing, pointer-reference, CopyFrom, and refNum goals. Done means the remaining dictionary-encoded struct copying is reduced without breaking the listed behaviors.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
performance
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.