infiniflow / infiniflow/infinity

[Feature Request]: United resource control for inverted index

Open
#550 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
4.7k
Forks
445
Avg merge
2d 2h
Merged PRs (30d)
7

Description

1. Using global thread pool or task scheduler for index building. Currently, a task executor is used at a columnar granularity, which is lack of being aware of global available threads.
2. Global memory quota control together with columnar store
3. Global segment id allocator

Contributor guide

Open the contributing guide

Research direction

Start by mapping the inverted-index index-building path and its columnar-granularity task executor. Then examine how the columnar store manages memory and how segment IDs are allocated. Done means defining and implementing a coordinated approach for global thread scheduling, memory quotas, and segment ID allocation, with behavior verified across these paths.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases, search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.