github / github/rust-gems

Performance improvement in `SimHashIterator`

未关闭
#78 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
Rust
星标
134
派生
24
平均合并
16 小时 27 分钟
30 天内合并 PR
11

描述

The current `SimHashIterator` works by finding the most significant active bit, and then calculating the number of `SimHash` buckets that will exist from that bit to the end of the filter. This means we are iterating `(most_significant_bit / SIM_BUCKET_SIZE) + SIM_BUCKETS` buckets.

The problem here is that in the higher order bits the sparsity is very high, meaning that there will potentially be many buckets where the values we're mixing into our `SimHash`es will just be zeros, and given any value XOR `0` is just that value this is wasted computation.

We could move to use the `BitChunks` iterators which batch up our chunks into a `u64` and an index of that block of bits, skipping ranges full of zeros. This will introduce issues where if our `SimHash` bucket straddles multiple `BitChunks`, which can be solved by peeking the next chunk, making sure that the index is contiguous with our current chunk.

Right now `SIM_BUCKET_SIZE` is always `6` bits but if we allowed fully arbitrary `SimHash` bucket sizes this could get slightly gnarly because we'd need to be able to peek an arbitrary number of `BitChunk`s ahead of the current position.

The following is a line chart plot of the proportion of empty `bit_range`s mixed into the SimHashes using pseudorandomly generated geofilters with `0..100_000` items added to them.

Image

The ratio reduces very sharply after a couple of inserts but remains relatively high since newly added items have some chance of landing in a very significant bucket which introduces a load of zero gaps before the next one bit.

贡献指南

打开贡献指南

调研方向

首先在仓库中找到 SimHashIterator、BitChunks 和 SIM_BUCKET_SIZE,然后阅读当前 bucket 的形成方式。研究 BitChunks 如何跳过零值范围,以及如何处理跨越多个 chunk 的 bucket。在不改变 SimHash bucket 结果的情况下减少空范围混合,即视为完成。

由索引模型根据 Issue 内容生成。

评估

技术栈
rust
领域
performance
Issue 类型
重构
难度
4/5
预计耗时
3-5 天
活跃度
停滞
描述清晰度
基本清楚
新手友好度
38/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。