Observer cores during async vector index processing due to invalid VSAG parameters and schema fallback

Environment

• Version: observer (OceanBase seekdb)
• Source Revision: 1-6fe7d0399a386908aa5bd18bdf30e27850a05a5c
• Build Branch: master
• Build Time: Jul 3 2026 23:31:50
• Build Flags: RelWithDebInfo|Sanity
• Build Info: obbuild-sanity-master-101220
• Is RD-provided temporary version: No
• Reproducibility: Deterministic (必现)
• Test changes: Not specified
• Other environmental anomalies/prompts: No

Description

During stress testing (stress_duomo_lite), the observer process crashed with a SIGABRT due to a memory_sanity_abort. The crash occurred in the T1_CSWorker220 thread while processing an asynchronous hybrid/vector index.

The root cause is twofold:

1. Triggering Condition: The workload generated an illegal VSAG parameter combination (m=22, ef_construction=30) for a table containing an asynchronous hybrid/vector index. The internal max_degree is calculated as 44, making ef_construction=30 invalid as it must be >= max_degree.
2. Core Failure: The observer's error handling for this scenario is unstable. The illegal parameter should be rejected with an error (e.g., during table creation) rather than causing a cascade of errors (async index worker failures, schema version mismatch, fallback schema manager rebuild) that ultimately leads to a memory sanity abort and crash during a schema deep copy operation.

Steps to Reproduce

The issue can be reproduced by creating a table with specific illegal VSAG parameters for an async vector index and then performing operations that trigger the CS async index processing.

1. Create Table with Illegal Parameters

CREATE DATABASE IF NOT EXISTS test;
USE test;

DROP TABLE IF EXISTS t_vec_async_repro;

CREATE TABLE t_vec_async_repro (
  seed INT NOT NULL PRIMARY KEY,
  v1 VECTOR(3),
  VECTOR INDEX idx1(v1)
    WITH(distance=l2,type=hnsw_sq,lib=vsag,m=22,ef_construction=30),
  hybrid_vector_doc VARCHAR(100),
  VECTOR INDEX hybrid_vector_idx(hybrid_vector_doc)
    WITH(distance=l2,lib=vsag,type=hnsw_bq,model=ob_embed,dim=1024,
         sync_mode=async,sync_interval=2343h)
) ORGANIZATION = HEAP;

2. Insert Data and Trigger CS Async Index

INSERT INTO t_vec_async_repro(seed, v1, hybrid_vector_doc)
VALUES
  (1, '[1,2,3]', 'doc-a'),
  (2, '[2,3,4]', 'doc-b'),
  (3, '[3,4,5]', 'doc-c');
ALTER TABLE t_vec_async_repro ADD COLUMN c1 INT;
ALTER TABLE t_vec_async_repro DROP COLUMN c1;
ALTER TABLE t_vec_async_repro ADD COLUMN c2 CHAR(16);
ALTER TABLE t_vec_async_repro DROP COLUMN c2;

Impact

• Observer process crash (SIGABRT, core dump).
• Service interruption for the affected tenant/database.
• The crash is triggered by a specific, illegal parameter combination in vector index creation, which workload generation tools might produce.

Logs & Core Analysis

• Core File: /data/1/bak_core/20260706_145309/core-T1_CSWorker220-2699203-1783320782
• Crashing Process: /data/1/vec_stress/duomo_merged_4h/duomomerged.z1.obs0/bin/observer (PID: 2699203)
• Signal: SIGABRT
• Crashing Thread: T1_CSWorker220
• Call Stack: The crash occurs in the path:
  ObCSPluginAsyncIndex -> ObCSAsyncIndexProcessor::init_schema_guard_ -> ObMultiVersionSchemaService::construct_fallback_schema_mgr_ -> ObSchemaMgr::deep_copy/add_table -> ObPointerHashMap -> memory_sanity_abort
• Error Context:
  • Trace ID YB427F000001-000655EB9B7A53BE-0-0 shows a vector query on table ddlkitty_1783319874927194 returned 4016 Internal error at 14:41:10.
  • The table was successfully created at 14:38:22 (see ddl_hybrid.log).
  • At 14:53:01, CSWorker processing the async vector index encountered VSAG rejection: ef_construction=30, max_degree=44 invalid vsag parameter Failed to lazily init vsag for async index.
• Database Logs: Located at /data/1/vec_stress/duomo_merged_4h/duomomerged.z1.obs0/log
• Jenkins Job: http://:9090/jenkins/job/stress_duomo_lite/
• Work ID: xJMcKDVRoXpa89uA

Proposed Solution

1. Parameter Validation: Add validation during table/index creation to reject illegal VSAG parameter combinations (e.g., ef_construction < max_degree). Return a clear error to the user.
2. Error Handling Robustness: Improve the error handling in the async index processing and schema fallback paths to prevent a single parameter error from cascading into a schema manager corruption and subsequent crash. The system should handle the failure gracefully, perhaps by marking the specific index as failed and logging an error, without bringing down the observer.

Priority

High. This is a deterministic crash caused by invalid but plausible user input, leading to service unavailability.

Start with the SQL reproduction and trace the reported path from ObCSPluginAsyncIndex through ObCSAsyncIndexProcessor::init_schema_guard_ and construct_fallback_schema_mgr_. Review the VSAG rejection and schema deep-copy logs to define validation and failure handling. Done means invalid parameters are rejected clearly and async processing no longer causes an observer crash.