bytedance / bytedance/videx

Validate the Performance and Accuracy of Sampling-based Metadata Collection

Open
#69 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
149
Forks
27
PR merge metrics
No merged PRs in 30d

Description

## Background
The sampling-based metadata collection feature was introduced in PR #48 as a faster alternative to full-table scans for gathering statistics (NDV, histograms). While this feature is intended to reduce the performance impact of data collection on production systems, it has not yet been thoroughly tested or validated.

This issue tracks the work required to benchmark its performance and validate the accuracy of the statistics it produces.

## Task Description
The validation process should cover three key areas:

1. **Accuracy Validation**:
* For various table sizes and data distributions (e.g., from TPC-H and JOB benchmarks), collect statistics using both sampling and a full scan.
* Compare the generated statistics (NDV, histogram bucket boundaries, frequencies) to quantify the error/deviation introduced by sampling.

2. **Performance Benchmarking**:
* Measure and compare the wall-clock time required for metadata collection using sampling versus a full scan across different table sizes.
* Document the performance gains.

3. **End-to-End Plan Validation**:
* Generate two sets of VIDEX metadata for the TPC-H and JOB benchmarks: one with full-scan statistics and one with sampled statistics.
* Run the full set of benchmark queries against VIDEX using both metadata sets.
* Compare the resulting query plans to identify any significant regressions caused by inaccuracies in the sampled statistics.
* Compare the plans generated using sampled stats against the plans from a native database instance.

#### Acceptance Criteria
- [ ] Accuracy and performance tests are executed on both TPC-H and JOB benchmarks.
- [ ] A summary of the findings is documented in a report or comment on this issue. The report should detail the trade-offs between collection speed and statistical accuracy.
- [ ] Any significant plan regressions or bugs discovered during validation are filed as new, separate issues.

Contributor guide

No contributing guide indexed for this repository

Research direction

Review PR #48 and the sampling-based metadata collection entry points before planning the validation. Run accuracy and wall-clock comparisons for sampled versus full-scan statistics on the TPC-H and JOB benchmarks, then compare VIDEX query plans with both metadata sets and a native database instance. Done means a documented trade-off report and separate issues for any significant regressions or bugs.

Written by the indexing model from the issue text.

Assessment

Tech stack
mariadb, mysql, python
Domain
databases, performance, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.