Performance improvement in `SimHashIterator`
- 主要语言
- Rust
- 星标
- 134
- 派生
- 24
- 平均合并
- 16 小时 27 分钟
- 30 天内合并 PR
- 11
描述
The current `SimHashIterator` works by finding the most significant active bit, and then calculating the number of `SimHash` buckets that will exist from that bit to the end of the filter. This means we are iterating `(most_significant_bit / SIM_BUCKET_SIZE) + SIM_BUCKETS` buckets.
The problem here is that in the higher order bits the sparsity is very high, meaning that there will potentially be many buckets where the values we're mixing into our `SimHash`es will just be zeros, and given any value XOR `0` is just that value this is wasted computation.
We could move to use the `BitChunks` iterators which batch up our chunks into a `u64` and an index of that block of bits, skipping ranges full of zeros. This will introduce issues where if our `SimHash` bucket straddles multiple `BitChunks`, which can be solved by peeking the next chunk, making sure that the index is contiguous with our current chunk.
Right now `SIM_BUCKET_SIZE` is always `6` bits but if we allowed fully arbitrary `SimHash` bucket sizes this could get slightly gnarly because we'd need to be able to peek an arbitrary number of `BitChunk`s ahead of the current position.
The following is a line chart plot of the proportion of empty `bit_range`s mixed into the SimHashes using pseudorandomly generated geofilters with `0..100_000` items added to them.
The ratio reduces very sharply after a couple of inserts but remains relatively high since newly added items have some chance of landing in a very significant bucket which introduces a load of zero gaps before the next one bit.
贡献指南
调研方向
首先在仓库中找到 SimHashIterator、BitChunks 和 SIM_BUCKET_SIZE,然后阅读当前 bucket 的形成方式。研究 BitChunks 如何跳过零值范围,以及如何处理跨越多个 chunk 的 bucket。在不改变 SimHash bucket 结果的情况下减少空范围混合,即视为完成。
由索引模型根据 Issue 内容生成。
评估
- 技术栈
- rust
- 领域
- performance
- Issue 类型
- 重构
- 难度
- 4/5
- 预计耗时
- 3-5 天
- 活跃度
- 停滞
- 描述清晰度
- 基本清楚
- 新手友好度
- 38/100