pingcap / pingcap/tiflash

Support rough set filter for Decimal columns

Open
#11,031 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

affects-8.5
Dominant language
C++
Stars
1k
Forks
423
Avg merge
1d 15h
Merged PRs (30d)
24

Description

Background

TiFlash rough set filter can use storage-level min/max metadata to skip packs before reading full column data. Decimal columns are common for monetary and metric fields, but decimal predicates are currently not handled end-to-end by the rough set filter path, so queries with selective Decimal filters cannot benefit from pack pruning.

For example:

SELECT *
FROM t
WHERE decimal_col >= 100.00 AND decimal_col < 200.00;

Without Decimal rough set support, TiFlash may need to scan more packs even when the Decimal predicate is highly selective.

Proposal

Support Decimal columns in the rough set min/max filter path.

Main scope:

  • Build and persist min/max index metadata for Decimal columns.
  • Read Decimal min/max metadata from DMFile and expose it to rough set checks.
  • Allow Decimal column predicates to be parsed into rough set operators only when the non-null literal is encoded as tipb::ExprType::MysqlDecimal.
  • Keep mixed Decimal/non-Decimal predicates conservative: parser should mark them as unsupported when statically detectable, and runtime CanNotCompare -> Some should remain as a safety net.
  • Use scale-aware Decimal comparison for Decimal32/Decimal64/Decimal128/Decimal256, including different precision/scale combinations between the predicate constant and the column/index value.
  • Support comparison predicates, null-safe equality, IN, and nullable Decimal columns where applicable.

Acceptance Criteria

  • Decimal predicates can create effective rough set operators and prune packs with Decimal min/max indexes.
  • Mixed Decimal/non-Decimal predicates do not create misleading rough set operators and remain conservative.
  • Unit tests cover parser behavior, min/max rough check behavior, nullable Decimal behavior, and DMFile write/read of Decimal min/max indexes.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the rough set parser and checks, then trace DMFile min/max metadata write and read paths for Decimal columns. Add coverage for parser behavior, scale-aware comparisons, nullable values, mixed-type predicates, and Decimal min/max persistence. Done means selective Decimal predicates prune packs while unsupported mixed predicates remain conservative.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.