Add Vector Search API support

Jira task: https://scylladb.atlassian.net/browse/DRIVER-733
Jira epic: https://scylladb.atlassian.net/browse/DRIVER-729

Copied from Jira epic DRIVER-729:

This issue tracks client-library support for the Alternator vector search API.

Starting point / reference implementation: https://github.com/scylladb/alternator-client-java/pull/87

This is distinct from DRIVER-112, which tracks CQL/native-driver vector support. Alternator vector search is a DynamoDB-compatible API extension, so standard AWS SDK models do not know the extra request/response fields and may reject or drop them unless the client library patches the model or intercepts raw JSON.

Core/server references

• Initial core issue / implementation PR: https://github.com/scylladb/scylladb/pull/29046
• Merge commit for initial support: https://github.com/scylladb/scylladb/commit/37fc1507f0798a3eae14c6a17aa7370f9155c92c
• More features: https://github.com/scylladb/scylladb/pull/29554
• Pre-filtering, Limit <= 1000, KeyConditions rejection: https://github.com/scylladb/scylladb/pull/29776
• Projected columns for vector indexes: https://github.com/scylladb/scylladb/pull/29959
• Vector search and Alternator Streams coexistence: https://github.com/scylladb/scylladb/pull/29826

Related vector-store references:

• https://github.com/scylladb/vector-store/pull/392
• https://github.com/scylladb/vector-store/pull/394
• https://github.com/scylladb/vector-store/pull/444

Server docs/tests:

• https://github.com/scylladb/scylladb/blob/master/docs/alternator/vector-search.md
• https://github.com/scylladb/scylladb/blob/master/test/alternator/test_vector.py

Required client API surface

CreateTable.VectorIndexes

Support vector index definitions with IndexName, VectorAttribute, optional Projection, and optional SimilarityFunction.

VectorAttribute contains AttributeName and Dimensions. SimilarityFunction values are COSINE, EUCLIDEAN, and DOT_PRODUCT; COSINE is the server default.

Projection follows the DynamoDB secondary-index projection shape: ProjectionType plus optional NonKeyAttributes. Current merged server support is KEYS_ONLY. Projection=INCLUDE is being added by scylladb/scylladb#29959. Projection=ALL is not currently supported.

Useful client-side validation: Dimensions must be positive and the server maximum is 16000; IndexName uses DynamoDB-style table/index name rules; the vector attribute cannot be a table key or secondary-index key; duplicate vector index names and duplicate vector indexes on the same attribute are invalid.

UpdateTable.VectorIndexUpdates

Support VectorIndexUpdates with exactly one operation per request: Create or Delete. Create has the same shape as CreateTable.VectorIndexes. Delete contains IndexName. There is no vector-index Update operation.

Do not combine vector index updates with GSI updates in the same request. After scylladb/scylladb#29826, vector indexes and Alternator Streams can coexist on the same table, but stream status changes and vector index create/delete should not be combined in one UpdateTable request.

DescribeTable.Table.VectorIndexes

Expose VectorIndexes from DescribeTable. Returned fields include IndexName, VectorAttribute, Projection, SimilarityFunction, IndexStatus, and Backfilling. Client libraries should provide or document a waiter/helper that waits for IndexStatus == ACTIVE before vector queries.

AttributeValue.FLOAT32VECTOR

Support the Alternator-specific AttributeValue member FLOAT32VECTOR, for example:

{"FLOAT32VECTOR": [0.1, -0.3, 0.7]}

Values are JSON numbers, not DynamoDB numeric strings. Values must fit finite float32. The vector length must match the index dimensions. Standard DynamoDB L-of-N vectors are also accepted by the server, but FLOAT32VECTOR is the compact optimized representation.

Most AWS SDKs do not know FLOAT32VECTOR. A client library may need to patch the SDK service model or intercept serialized request/response JSON. If a local placeholder representation is used internally, it must be replaced with real FLOAT32VECTOR JSON before sending the request.

On reads, preserve enough information for callers to re-write the value as FLOAT32VECTOR. If the client converts it only to a normal L-of-N value, copying the item back will store it as a regular list instead of the compact vector type.

Query.VectorSearch

Support Query.VectorSearch with QueryVector and optional ReturnScores.

Query rules: IndexName is required and must name a vector index. Limit is required and must be positive. scylladb/scylladb#29776 caps Limit at 1000. VectorSearch.QueryVector is required and may be FLOAT32VECTOR or L-of-N. ConsistentRead=true is rejected. ExclusiveStartKey is rejected because there is no pagination. ScanIndexForward is rejected because results are always nearest/best first. Legacy QueryFilter is rejected.

ReturnScores values are NONE and SIMILARITY. NONE is the default. With SIMILARITY, the response includes Scores, a numeric array parallel to Items where Scores[i] belongs to Items[i]. ReturnScores=SIMILARITY is invalid with Select=COUNT.

Score semantics: higher is better. COSINE scores are in [0, 1] using (1 + cosine) / 2. EUCLIDEAN scores are in [0, 1] using 1 / (1 + distance). DOT_PRODUCT is unbounded and may be negative or greater than 1.

Standard Query options: Select, ProjectionExpression, AttributesToGet, and FilterExpression are supported. FilterExpression is a post-filter and can return fewer than Limit results. The current default Select behavior is ALL_PROJECTED_ATTRIBUTES. Select=COUNT returns counts and no Items.

Filtering caveat: current merged server behavior ignores KeyConditionExpression for vector search. scylladb/scylladb#29776 changes KeyConditionExpression into a pre-filter over projected attributes and rejects legacy KeyConditions. Supported pre-filter operators after that PR are =, <, <=, >, >=, IN, BETWEEN, and AND. OR, NOT, and <> are not supported. The pre-filter is limited to projected scalar attributes, currently S and N.

Implementation guidance

• Preserve normal DynamoDB behavior for requests that do not use Alternator vector extensions.
• Prefer a typed, language-idiomatic API, but keep a raw escape hatch for unsupported or not-yet-modeled server fields.
• If the underlying SDK drops unknown response fields, extract VectorIndexes, Scores, and FLOAT32VECTOR before normal SDK parsing.
• Integration tests should skip cleanly when the server does not support vector indexes or when vector-store is disabled.

Acceptance criteria

• CreateTable.VectorIndexes can be serialized and sent.
• UpdateTable.VectorIndexUpdates create/delete can be serialized and sent.
• DescribeTable.Table.VectorIndexes can be parsed and exposed.
• FLOAT32VECTOR can be written and read without being lost or silently converted into an incompatible representation.
• Query.VectorSearch works with FLOAT32VECTOR and L-of-N query vectors.
• ReturnScores=SIMILARITY is exposed and Scores order matches Items order.
• Invalid combinations are covered: missing IndexName, missing Limit, missing QueryVector, ConsistentRead=true, ExclusiveStartKey, ScanIndexForward, and ReturnScores=SIMILARITY with Select=COUNT.
• Documentation/examples explain server-version caveats for KeyConditionExpression pre-filtering, Projection=INCLUDE, Limit <= 1000, and Streams coexistence.

Start by comparing the reference implementation in alternator-client-java/pull/87 with docs/alternator/vector-search.md and test/alternator/test_vector.py. Map the required CreateTable, UpdateTable, DescribeTable, AttributeValue, and Query surfaces onto this Java client and identify how unknown JSON is handled. Done means the listed vector operations, FLOAT32VECTOR preservation, validation cases, scores, and server-version caveats are covered by client behavior and tests.