spring-projects / spring-projects/spring-ai

OpenSearchVectorStore Does Not Fully Apply Mapping Configuration During Index Initialization

Open
#2,985 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

status: waiting-for-triage
Dominant language
Java
Stars
9.5k
Forks
2.9k
Avg merge
1d 7h
Merged PRs (30d)
6

Description

When using Spring AI's (version: 1.0.0-M6) OpenSearchVectorStore, the custom mapping configuration is not fully applied during index initialization. Specifically:

loadCustomMapping:
{
"mappings": {
"properties": {
"id": {
"type": "keyword"
},
"content": {
"type": "text",
"analyzer": "standard"
},
"metadata": {
"type": "object"
},
"embedding": {
"type": "knn_vector",
"dimension": 1536,
"method": {
"name": "hnsw",
"space_type": "cosinesimil",
"engine": "lucene"
}
}
}
},
"settings": {
"index": {
"knn": true,
"number_of_shards": 1,
"number_of_replicas": 0
}
}
}

@Bean
public VectorStore vectorStore(OpenSearchClient openSearchClient, EmbeddingModel embeddingModel) {
return OpenSearchVectorStore.builder(openSearchClient, embeddingModel)
.index(indexName)
.mappingJson(loadCustomMapping())
.similarityFunction(similarityFunction)
.initializeSchema(initializeSchema)
.batchingStrategy(new TokenCountBatchingStrategy())
.build();
}

  1. During initial index creation, only basic settings are applied:
    "settings": {
    "index": {
    "replication": {
    "type": "DOCUMENT"
    },
    "number_of_shards": "1",
    "provided_name": "mem,
    "knn": "true",
    "creation_date": "1746330816561",
    "number_of_replicas": "1",
    "uuid": "43LTNIlUT8Stt9dADOyqmA",
    "version": {
    "created": "136408027"
    }
    }
    }
  2. After inserting the first document, the embedding field is automatically inferred as float type, instead of the knn_vector type specified in our mapping.json:
"embedding": {
  "type": "float"
}
  1. Our expected configuration (mapping.json) is:
{
  "mappings": {
    "properties": {
      "id": {
        "type": "keyword"
      },
      "content": {
        "type": "text",
        "analyzer": "standard"
      },
      "metadata": {
        "type": "object"
      },
      "embedding": {
        "type": "knn_vector",
        "dimension": 1536,
        "method": {
          "name": "hnsw",
          "space_type": "cosinesimil",
          "engine": "lucene"
        }
      }
    }
  },
  "settings": {
    "index": {
      "knn": true,
      "number_of_shards": 1,
      "number_of_replicas": 0
    }
  }
}

The mapping.json configuration should be fully applied when creating the index, specifically:

  1. The embedding field should be created as knn_vector type
  2. All KNN-related settings should be applied
  3. Shards and replicas should be set according to the configuration

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the OpenSearchVectorStore entry point and trace how mappingJson is handled during index initialization. Compare the requested mappings and settings with the created index and first document behavior. Done means the configured knn_vector mapping, KNN settings, shard count, and replica count are applied when the index is created.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.