[VI] Requirements gathering

Goal

The goal of this task is to gather and document the requirements for implementing native vector indexing in MySQL. It will capture expected SQL capabilities, supported vector operations and search behavior, transactional and MVCC guarantees, performance and scalability expectations, operability needs, compatibility considerations, and integration requirements across InnoDB, replication, backup, recovery, security, and observability.

SQL Syntax

Define consistent and extensible SQL syntax for creating, altering, rebuilding, and dropping vector indexes, as well as performing similarity searches. The syntax should support distance measures, algorithm selection and index-specific options while remaining compatible with existing MySQL conventions.

The syntax proposals are being discussed here : Syntax Discussion

Native Default algorithm

The project will deliver one native vector indexing algorithm as the default implementation. It will also establish a robust, extensible architectural foundation that allows additional algorithms, including non-default implementations, to be integrated in the future through well-defined interfaces.

ACID compliant

The vector index must be fully integrated with MySQL transaction processing and preserve all ACID guarantees. Index changes must participate in the same transaction as the corresponding table changes, with proper redo logging for durability and crash recovery, and undo logging for rollback and MVCC consistency.

Configuration/Tuning requirements

Identify the configuration parameters required to control vector index creation, maintenance, and search behavior. The implementation should provide reasonable defaults while allowing users to tune algorithm-specific settings, memory usage, parallelism, accuracy-versus-latency trade-offs, and background maintenance according to their workloads.

Observability

The implementation should provide sufficient metrics, diagnostics, and metadata to monitor vector index health, usage, performance, memory and storage consumption, build and maintenance status, and query behavior. This information should be exposed through standard MySQL interfaces such as Performance Schema, Information Schema, EXPLAIN, and server logs.

Performance consideration

Define representative workloads, datasets, metrics, and baseline benchmarks for evaluating the vector index implementation. Testing should measure index build time, query latency and throughput, recall, DML overhead, memory and storage usage, scalability, and performance under concurrent transactional workloads.

Future Extensibility

The architecture should provide clearly defined, stable interfaces that allow new vector indexing algorithms to be added without significant changes to MySQL or InnoDB core components. These interfaces should enable independent implementations to integrate with index creation, maintenance, search, transaction processing, persistence, recovery, and observability.

Anything else?

Start by reading the requirements in this issue and the linked Syntax Discussion in issue #702. Produce a consolidated requirements document covering SQL syntax, indexing behavior, transactions, configuration, observability, performance benchmarks, and extensibility, with agreement on the expected scope as the definition of done.