oceanbase / oceanbase/seekdb

Fix incorrect NDV (1) for VARCHAR columns in DBA_TAB_COL_STATISTICS during online statistics gathering with online_osg_sample / ONLINE_ESTIMATE_PERCENT

Open
#725 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

dima type: bug
Dominant language
C++
Stars
2.9k
Forks
339
Avg merge
1d 19h
Merged PRs (30d)
52

Description

Description

During online statistics gathering (specifically in the online_osg_sample test case related to the ONLINE_ESTIMATE_PERCENT preference), the NDV (Number of Distinct Values) for VARCHAR columns in the DBA_TAB_COL_STATISTICS view is incorrectly reported as 1.

Actual Behavior:
The NDV for VARCHAR columns is 1.

Expected Behavior:
The NDV should reflect the actual number of distinct values in the column.

Steps to Reproduce

  1. Log into the test machine.

  2. Navigate to the directory: /obdata/data/AndroidTest/ob_opensource_test/tools/deploy.

  3. Execute the following commands to set up the test environment and run the specific mysqltest case:

    export OCEANBASE_HOME=/obdata/data/AndroidTest/ob_opensource_test/
    cd ${OCEANBASE_HOME}/tools/deploy
    source activate_obd.sh
    export OBD_SEEKDB_STATUS_FALLBACK_TCP=1
    
    obd test mysqltest test_env --mysqltest-bin=${OCEANBASE_HOME}/deps/3rd/u01/obclient/bin/mysqltest --obclient-bin=${OCEANBASE_HOME}/deps/3rd/u01/obclient/bin/obclient --init-only --init-sql-dir=${OCEANBASE_HOME}/tools/deploy --init-sql-files='init.sql,init_user.sql|root@sys|test'
    obd test mysqltest test_env --mysqltest-bin=${OCEANBASE_HOME}/deps/3rd/u01/obclient/bin/mysqltest --obclient-bin=${OCEANBASE_HOME}/deps/3rd/u01/obclient/bin/obclient --disable-reboot --test-set=online_opt_stat_gather.online_osg_sample
    
  4. The test case executes the following SQL sequence which reproduces the issue:

    USE test;
    
    DROP TABLE IF EXISTS t1, t2, t3, t4, t5;
    DROP PROCEDURE IF EXISTS batch_insert;
    
    CREATE TABLE t1 (c1 BIGINT PRIMARY KEY, c2 VARCHAR(256), c3 VARCHAR(256), c4 DATETIME, c5 DATETIME)
      WITH COLUMN GROUP (each column);
    CREATE TABLE t2 LIKE t1;
    CREATE TABLE t3 LIKE t1;
    CREATE TABLE t4 LIKE t1;
    CREATE TABLE t5 LIKE t1;
    
    DELIMITER $$
    CREATE PROCEDURE batch_insert(IN args INT)
    BEGIN
      DECLARE i INT;
      SET i = 0;
      START TRANSACTION;
      WHILE i < args DO
        INSERT INTO t1(c1,c2,c3,c4,c5) VALUES (
          i,
          CONCAT('C2_MKKHJHHS_XXXSDSDSDDSDSSJJJJDSHH_', i),
          CONCAT('C3_MKKHJHHS_XXXMMMMMMJJJDSHH_', i),
          '2023-01-01 01:11:11',
          '2023-01-01 01:11:11'
        );
        SET i = i + 1;
        IF i % 10000 = 0 THEN COMMIT; END IF;
      END WHILE;
      COMMIT;
    END$$
    DELIMITER ;
    
    CALL batch_insert(100000);
    
    CALL dbms_stats.set_global_prefs('ONLINE_ESTIMATE_PERCENT', '1');
    INSERT /*+ query_timeout(100000000) append parallel(2) enable_parallel_dml */ INTO t2 SELECT c1, c2, c3, c4, c5 FROM t1;
    INSERT /*+ query_timeout(100000000) append parallel(2) enable_parallel_dml */ INTO t3 SELECT c1, c2, c3, c4, c5 FROM t1;
    
    CALL dbms_stats.set_global_prefs('ONLINE_ESTIMATE_PERCENT', '50');
    INSERT /*+ query_timeout(100000000) append parallel(2) enable_parallel_dml */ INTO t4 SELECT c1, c2, c3, c4, c5 FROM t1;
    INSERT /*+ query_timeout(100000000) append parallel(2) enable_parallel_dml */ INTO t5 SELECT c1, c2, c3, c4, c5 FROM t1;
    
    CALL dbms_stats.set_global_prefs('ONLINE_ESTIMATE_PERCENT', '1');
    
    SELECT TABLE_NAME, NUM_ROWS, SAMPLE_SIZE
    FROM oceanbase.DBA_TAB_STATISTICS
    WHERE table_name IN ('t2','t3','t4','t5')
    ORDER BY table_name;
    
    SELECT TABLE_NAME, COLUMN_NAME, NUM_DISTINCT, LOW_VALUE, HIGH_VALUE
    FROM oceanbase.DBA_TAB_COL_STATISTICS
    WHERE table_name IN ('t2','t3','t4','t5')
    ORDER BY table_name, column_name;
    

The final queries will show that the NUM_DISTINCT for the VARCHAR columns (c2, c3) is 1, which is incorrect.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the online_opt_stat_gather.online_osg_sample mysqltest case with the supplied setup commands, then inspect the online statistics gathering implementation for VARCHAR NDV handling. Confirm the fix by querying oceanbase.DBA_TAB_COL_STATISTICS for c2 and c3 in t2 through t5 and verifying NUM_DISTINCT reflects the inserted distinct values.

Written by the indexing model from the issue text.

Assessment

Tech stack
sql
Domain
databases, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.