quickwit-oss / quickwit-oss/quickwit
Tech debt: fix caching
Open
Nobody has claimed this yet.
project:airmail
- Dominant language
- Rust
- Stars
- 11.7k
- Forks
- 597
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 37
Description
In #680 we replugged the long term cache, caching fast fields.
The solution is however not great as
- it relies on the fact that segment ids are universally unique
- it uses a global.
It would be worth putting more thought on how this should be implemented.
/// Wraps the given directory with a slice cache that is actually global
/// to quickwit.
///
/// FIXME The current approach is quite horrible in that:
/// - it uses a global
/// - it relies on the idea that all of the files we attempt to cache
/// have universally unique names. It happens to be true today, but this might be very error prone
/// in the future.
pub fn wrap_storage_with_long_term_cache(storage: Arc<dyn Storage>) -> Arc<dyn Storage> {
static SINGLETON: OnceCell<Arc<dyn Cache>> = OnceCell::new();
let cache = SINGLETON
.get_or_init(|| Arc::new(QuickwitCache::default()))
.clone();
Arc::new(StorageWithCache { storage, cache })
}
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the wrap_storage_with_long_term_cache function shown in the issue and review the caching change from #680. Determine a design that avoids the global cache and does not depend on universally unique segment IDs; done means the caching approach has been redesigned around those concerns.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, performance
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100