colinmarc / colinmarc/cdb

Add support for `mmap`-backed CDB reader (yielding a x35 improvement)

Open
#11 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
69
Forks
13
PR merge metrics
No merged PRs in 30d

Description

Currently the CDB reader uses `io.ReaderAt`, that for each read operation will issue a `read` system call (or equivalent). An alternative would be to use `mmap`-ed memory to access the backing file.

In my experiments (see the patch below) by using `mmap` yields a 35 times performance improvement:

~~~~
BenchmarkGet-4 13902710 2571 ns/op 28 B/op 2 allocs/op
BenchmarkGetMmap-4 484713384 73.96 ns/op 0 B/op 0 allocs/op
~~~~

The following are a few patches that provide such support. If accepted I can submit a pull request:

* https://github.com/cipriancraciun/go-cdb-lib/commit/6c948e58f34ad9eef36c4e94634f16985f76e231 -- adds support of using a `[]byte` buffer that backs the CDB reader; such a `[]byte` buffer can be either the result of a `mmap` or just reading the entire file in memory;
* https://github.com/cipriancraciun/go-cdb-lib/commit/0b1bb739d8fe8183d84e0cdb3f7b2cab996ede35 -- this adds support for a custom `io.Close` to be called on `cdb.Close`; (it will be used to `munmap` the memory if needed;)
* https://github.com/cipriancraciun/go-cdb-lib/commit/924564bd03fc301a773007971120026f8f1a0bf6 -- this adds the actual `mmap` support;

The only semantic change is the following:
* the `[]byte` value returned as result of `Get` should not be written to by the application, as it's actually a slice from the `mmap`-ed file;
* the `[]byte` value can't be used after the `cdb.Close` was called;
* if this is not acceptable, and one still wants to keep the current (undocumented) semantic, one could clone that slice (thus incurring an extra alloc, which in my patch doesn't happen);

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing the three referenced commits: the []byte-backed reader, custom io.Close handling, and mmap support. Compare their behavior with the current CDB reader and the BenchmarkGet results. Done means mmap-backed reads work, cleanup occurs on cdb.Close, and the documented Get value lifetime and write restrictions are resolved.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.