pingcap / pingcap/tiflash

ERROR 9014 (HY000): TiFlash backfill index failed: Code: 0

Open
#10,617 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

contribution type/bug
Dominant language
C++
Stars
1k
Forks
423
Avg merge
1d 15h
Merged PRs (30d)
24

Description

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)
  1. minio server /tmp/tidb
  2. tiup playground
    tiup playground nightly --mode=tidb-cse --tag cse-test
    --db.binpath=${tidb-server-path}
    --kv.binpath=${tikv-server-path}
    --pd.binpath=${pd-server-path}
    --tiflash.binpath=${tiflash-server-path} --tiflash.compute 1 --tikv.columnar
  3. create vector table
CREATE DATABASE IF NOT EXISTS vector_db_test;
USE vector_db_test;
mysql> show create table vector_bench_test\G
*************************** 1. row ***************************
       Table: vector_bench_test
Create Table: CREATE TABLE `vector_bench_test` (
  `id` bigint(20) NOT NULL /*T![auto_rand] AUTO_RANDOM(5) */,
  `embedding` vector(768) NOT NULL,
  PRIMARY KEY (`id`) /*T![clustered_index] CLUSTERED */,
  VECTOR INDEX `idx_embedding`((VEC_COSINE_DISTANCE(`embedding`)))
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin
1 row in set (0.03 sec)
  1. Vectorbench for inital dataset and query test.
2. What did you expect to see? (Required)

No error output, Vector Index can be used on the Vector Search.

3. What did you see instead (Required)
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+
| TIDB_DATABASE | TIDB_TABLE        | TIDB_PARTITION | TABLE_ID | BELONGING_TABLE_ID | COLUMN_NAME | INDEX_NAME   | COLUMN_ID | INDEX_ID | INDEX_KIND | ROWS_STABLE_INDEXED | ROWS_STABLE_NOT_INDEXED | ROWS_DELTA_INDEXED | ROWS_DELTA_NOT_INDEXED | ERROR_MESSAGE                                                                                                                                                                                                                                                                               | TIFLASH_INSTANCE |
+---------------+-------------------+----------------+----------+--------------------+-------------+--------------+-----------+----------+------------+---------------------+-------------------------+--------------------+------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+
| vector_test   | docs              | NULL           |      143 |                 -1 | emb         | idx_emb      |         3 |        1 | Vector     |                   0 |                       0 |                  0 |                      0 |                                                                                                                                                                                                                                                                                             | store-c18476da   |
| vector_test   | vector_bench_test | NULL           |      150 |                 -1 | embedding   | vector_index |         2 |        1 | Vector     |                   0 |                 1000000 |                  0 |                      0 | Code: 0, e.displayText() = DB::Exception: cannot load checksum framed data from tiflash/s138/data/ks_1_t_150/dmf_73/2.dat (errno = 115): (while reading from DTFile: s3://s138/data/ks_1_t_150/dmf_73, column: 2:): while building stable index for index_ids=[1], e.what() = DB::Exception | store-c18476da   |
+---------------+-------------------+----------------+----------+--------------------+-------------+--------------+-----------+----------+------------+---------------------+-------------------------+--------------------+------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+
2 rows in set (0.03 sec)

mysql> alter table vector_bench_test add VECTOR INDEX vector_index((VEC_COSINE_DISTANCE(embedding)));
ERROR 9014 (HY000): TiFlash backfill index failed: Code: 0, e.displayText() = DB::Exception: cannot load checksum framed data from tiflash/s138/data/ks_1_t_154/dmf_99/2.dat (errno = 115): (while reading from DTFile: s3://s138/data/ks_1_t_154/dmf_99, column: 2:): while building stable index for index_ids=[3], e.what() = DB::Exception
mysql> alter table vector_bench_test add VECTOR INDEX vector_index((VEC_COSINE_DISTANCE(embedding)));


ERROR 9014 (HY000): TiFlash backfill index failed: Code: 0, e.displayText() = DB::Exception: cannot load checksum framed data from tiflash/s138/data/ks_1_t_154/dmf_100/2.dat (errno = 115): (while reading from DTFile: s3://s138/data/ks_1_t_154/dmf_100, column: 2:): while building stable index for index_ids=[4], e.what() = DB::Exception

4. What is your TiFlash version? (Required)
mysql> select * from cluster_info
    -> ;
+-----------------+------------------+------------------+-------------------------+------------------------------------------+----------------------+---------------------+-----------+
| TYPE            | INSTANCE         | STATUS_ADDRESS   | VERSION                 | GIT_HASH                                 | START_TIME           | UPTIME              | SERVER_ID |
+-----------------+------------------+------------------+-------------------------+------------------------------------------+----------------------+---------------------+-----------+
| tidb            | 172.23.0.2:4000  | 172.23.0.2:10080 | 7.5.6-serverless        | cd5a2c789dcafb31be003e7271da246508b5f2f7 | 2025-12-16T15:46:32Z | 13h58m46.123246514s |      1282 |
| pd              | 172.23.0.2:2379  | 172.23.0.2:2379  | 8.5.0-serverless        | 6e2a4d2024251e23168d1ce356eea19c05e10089 | 2025-12-16T15:46:12Z | 13h59m6.123256162s  |         0 |
| tikv            | 172.23.0.2:20160 | 172.23.0.2:20180 | 7.5.0                   | c85a029d3ae08685d89509e99204c7a6bb268b59 | 2025-12-16T15:46:14Z | 13h59m4.123256988s  |         0 |
| tiflash_write   | 172.23.0.2:3930  | 172.23.0.2:20292 | 8.5.0-cse-96-g9568c16e5 | 9568c16e597211acd9809d6c4429b09b1a0d6ffc | 2025-12-16T15:46:50Z | 13h58m28.12325777s  |         0 |
| tiflash_compute | 172.23.0.2:3931  | 172.23.0.2:20293 | 8.5.0-cse-96-g9568c16e5 | 9568c16e597211acd9809d6c4429b09b1a0d6ffc | 2025-12-16T15:46:50Z | 13h58m28.123258694s |         0 |
+-----------------+------------------+------------------+-------------------------+------------------------------------------+----------------------+---------------------+-----------+
5 rows in set (0.06 sec)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the provided minio server and tiup playground setup, create the vector table, load the initial dataset, and run the ALTER TABLE command to reproduce the TiFlash backfill failure. Inspect the TiFlash error output around the checksum-framed DTFile read; done means vector index backfill completes without ERROR 9014 and the index can be used for vector search.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, sql
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.