bytedance / bytedance/videx

[Initiative] Integrate Core Statistics Logic into the Plugin

Open
#72 0 comments 0 reactions 0 assignees View on GitHub
important
Dominant language
Python
Stars
149
Forks
27
PR merge metrics
No merged PRs in 30d

Description

### Proposal Summary

This proposal outlines a major architectural enhancement for VIDEX 0.3.0: moving the core, non-AI-based statistical gathering and processing logic from the standalone Python server directly into the database plugin.

The objective is to make the external Python server an optional component, required only for users leveraging advanced, AI-driven estimation models.

### Motivation and Impact

The current architecture requires deploying and managing a separate Python service, which adds complexity, resource overhead, and an additional point of failure, even for users who only need the standard heuristic-based statistics. By integrating this functionality directly into the database plugin (for MySQL/MariaDB/PostgreSQL), we can significantly streamline the deployment process.

This change will make VIDEX lighter, faster for standard operations, and easier to set up and maintain. It aligns with a "batteries-included" philosophy for core features, while preserving the crucial extensibility for advanced AI models. The impact is a vastly simplified architecture for the majority of users, reducing the operational burden and making VIDEX a more self-contained and robust tool.

### Technical Plan

1. **Logic Porting:** Re-implement the default heuristic-based algorithms for NDV and cardinality estimation (currently in Python's `VidexModelInnoDB`) in C++ within the plugin's source code.
2. **Internal Storage Mechanism:** Design and implement a mechanism for the plugin to store and manage statistical metadata. This will involve using either existing system tables or creating custom tables within a dedicated schema inside the database.
3. **Data Access Layer:** Create C++ functions within the plugin to read from and write to these new statistics tables, effectively replacing the HTTP calls to the Python server.
4. **Configuration Switch:** Introduce a new session or global variable (e.g., `videx_statistic_source`) that allows the user to toggle between the internal implementation (`'internal'`) and the external HTTP server (`'external'`). The default will be `'internal'`.
5. **Refactor Handler Calls:** Modify the handler interface functions to check this configuration variable and route requests for NDV/cardinality to either the new internal C++ functions or the existing HTTP client code, ensuring backward compatibility and extensibility.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by comparing Python's VidexModelInnoDB with the plugin source, then trace the existing handler interface functions and HTTP client path. Done means heuristic NDV and cardinality logic, statistics storage and access, the internal/external configuration switch, and backward-compatible routing are implemented for the supported databases.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, mariadb, mysql, postgresql, python
Domain
database
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.