Add support for `mmap`-backed CDB reader (yielding a x35 improvement)
- Dominant language
- Go
- Stars
- 69
- Forks
- 13
- PR merge metrics
- No merged PRs in 30d
Description
Currently the CDB reader uses `io.ReaderAt`, that for each read operation will issue a `read` system call (or equivalent). An alternative would be to use `mmap`-ed memory to access the backing file.
In my experiments (see the patch below) by using `mmap` yields a 35 times performance improvement:
~~~~
BenchmarkGet-4 13902710 2571 ns/op 28 B/op 2 allocs/op
BenchmarkGetMmap-4 484713384 73.96 ns/op 0 B/op 0 allocs/op
~~~~
The following are a few patches that provide such support. If accepted I can submit a pull request:
* https://github.com/cipriancraciun/go-cdb-lib/commit/6c948e58f34ad9eef36c4e94634f16985f76e231 -- adds support of using a `[]byte` buffer that backs the CDB reader; such a `[]byte` buffer can be either the result of a `mmap` or just reading the entire file in memory;
* https://github.com/cipriancraciun/go-cdb-lib/commit/0b1bb739d8fe8183d84e0cdb3f7b2cab996ede35 -- this adds support for a custom `io.Close` to be called on `cdb.Close`; (it will be used to `munmap` the memory if needed;)
* https://github.com/cipriancraciun/go-cdb-lib/commit/924564bd03fc301a773007971120026f8f1a0bf6 -- this adds the actual `mmap` support;
The only semantic change is the following:
* the `[]byte` value returned as result of `Get` should not be written to by the application, as it's actually a slice from the `mmap`-ed file;
* the `[]byte` value can't be used after the `cdb.Close` was called;
* if this is not acceptable, and one still wants to keep the current (undocumented) semantic, one could clone that slice (thus incurring an extra alloc, which in my patch doesn't happen);
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reviewing the three referenced commits: the []byte-backed reader, custom io.Close handling, and mmap support. Compare their behavior with the current CDB reader and the BenchmarkGet results. Done means mmap-backed reads work, cleanup occurs on cdb.Close, and the documented Get value lifetime and write restrictions are resolved.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100