redis / redis/redis-vl-python

SearchIndex.from_existing silently drops vector fields on a RESP2 connection with legacy_responses=False

Open
#712 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
427
Forks
101
Avg merge
6d 3h
Merged PRs (30d)
20

Description

Round-tripping an index through SearchIndex.from_existing and then create() rebuilds it without its vector field, silently, when the supplied client is a RESP2 connection created with redis-py 8's legacy_responses=False. Nothing is logged and nothing is raised. For a vector search library this is the worst available failure mode: the rebuilt index reports itself as healthy, FT.INFO lists it as built, and every vector query against it is searching for a field that no longer exists.

Reproduction

Measured on Redis 8.4.6 with redis-py 8.1.0, RedisVL at edcc78e. The source index has three fields, one of them a FLAT vector field named emb.

from redis import Redis
from redisvl.index import SearchIndex

client = Redis.from_url("redis://localhost:6379", protocol=2, legacy_responses=False)
recovered = SearchIndex.from_existing("my_index", redis_client=client)
print(sorted(recovered.schema.field_names))   # ['cat', 'text'], and 'emb' is gone

recovered.schema.index.name = "my_index_rebuilt"
recovered.create(overwrite=True)

FT.INFO my_index_rebuilt then reports attributes [b'text', b'cat']. The rebuilt index really does lack the vector field; that is a measured result rather than an inference from the schema.

Cause

redis-py's two unified FT.INFO parsers disagree about where vector parameters live. On a RESP3 connection they are promoted to top-level keys, measured on the same index as:

{b'identifier': b'emb', b'attribute': b'emb', b'type': b'VECTOR',
 b'algorithm': b'FLAT', b'data_type': b'FLOAT32', b'dim': 4,
 b'distance_metric': b'COSINE', b'flags': []}

On a RESP2 connection with legacy_responses=False, _parse_info_unified packs the same parameters into the flags list instead:

{'identifier': 'emb', 'attribute': 'emb', 'type': 'VECTOR',
 'flags': ['algorithm', 'FLAT', 'data_type', 'FLOAT32', 'dim', 4,
           'distance_metric', 'COSINE']}

parse_vector_attrs in redisvl/redis/connection.py takes its dict branch here and excludes flags from the keys it copies (line 321), so vector_attrs comes out empty. dims is therefore absent, and because dims is required the function returns None. The caller at line 548 reads that None as "vector attributes cannot be parsed on this Redis version" and does a bare continue at line 551, dropping the field from the schema without a word.

The continue is defensible on its own terms, since it was written for Redis 6.2.6-v9 where FT.INFO genuinely returns no vector parameters. What is not defensible is that it now also absorbs a parse failure on a modern server where the parameters were present in the reply all along.

Scope

Measured across three client configurations against the same index:

Client Fields recovered by from_existing
protocol=2 (RedisVL's own default) ['emb', 'text']
protocol=2, legacy_responses=False ['text']
protocol=3 ['emb', 'text']

Only the middle row is affected, so a fix should target the RESP2 unified shape rather than RESP3 handling, which already works. Note that legacy_responses is a redis-py 8 only keyword; on 6.3.0 and 7.4.0 it raises TypeError from AbstractConnection.__init__ at connect time, so this configuration is reachable only on redis-py 8.

Suggested fix

Two changes, independently useful. First, teach the dict branch of parse_vector_attrs to unpack a non-empty flags list as alternating key-value pairs before excluding it, which restores dims and the rest of the parameters. Second, replace the bare continue with a warning naming the field and the index, so that a future parse gap degrades loudly instead of silently. The second is worth doing regardless of the first: any path that yields a schema missing a field the server reported should say so.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in redisvl/redis/connection.py at parse_vector_attrs and the caller around lines 548-551; inspect the RESP2 unified FT.INFO shape and the existing vector parsing branches. Reproduce with Redis 8.4.6, redis-py 8.1.0, protocol=2, and legacy_responses=False. Done means from_existing preserves the emb vector field, create() rebuilds it, and unparseable fields produce a warning naming the field and index.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, redis
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.