[Initiative] Integrate Core Statistics Logic into the Plugin
- Dominant language
- Python
- Stars
- 149
- Forks
- 27
- PR merge metrics
- No merged PRs in 30d
Description
### Proposal Summary
This proposal outlines a major architectural enhancement for VIDEX 0.3.0: moving the core, non-AI-based statistical gathering and processing logic from the standalone Python server directly into the database plugin.
The objective is to make the external Python server an optional component, required only for users leveraging advanced, AI-driven estimation models.
### Motivation and Impact
The current architecture requires deploying and managing a separate Python service, which adds complexity, resource overhead, and an additional point of failure, even for users who only need the standard heuristic-based statistics. By integrating this functionality directly into the database plugin (for MySQL/MariaDB/PostgreSQL), we can significantly streamline the deployment process.
This change will make VIDEX lighter, faster for standard operations, and easier to set up and maintain. It aligns with a "batteries-included" philosophy for core features, while preserving the crucial extensibility for advanced AI models. The impact is a vastly simplified architecture for the majority of users, reducing the operational burden and making VIDEX a more self-contained and robust tool.
### Technical Plan
1. **Logic Porting:** Re-implement the default heuristic-based algorithms for NDV and cardinality estimation (currently in Python's `VidexModelInnoDB`) in C++ within the plugin's source code.
2. **Internal Storage Mechanism:** Design and implement a mechanism for the plugin to store and manage statistical metadata. This will involve using either existing system tables or creating custom tables within a dedicated schema inside the database.
3. **Data Access Layer:** Create C++ functions within the plugin to read from and write to these new statistics tables, effectively replacing the HTTP calls to the Python server.
4. **Configuration Switch:** Introduce a new session or global variable (e.g., `videx_statistic_source`) that allows the user to toggle between the internal implementation (`'internal'`) and the external HTTP server (`'external'`). The default will be `'internal'`.
5. **Refactor Handler Calls:** Modify the handler interface functions to check this configuration variable and route requests for NDV/cardinality to either the new internal C++ functions or the existing HTTP client code, ensuring backward compatibility and extensibility.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by comparing Python's VidexModelInnoDB with the plugin source, then trace the existing handler interface functions and HTTP client path. Done means heuristic NDV and cardinality logic, statistics storage and access, the internal/external configuration switch, and backward-compatible routing are implemented for the supported databases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, mariadb, mysql, postgresql, python
- Domain
- database
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100