opensearch-project / opensearch-project/flow-framework

[FEATURE] More variation for default use cases

Open
#667 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Java
Stars
62
Forks
66
Avg merge
3d 20h
Merged PRs (30d)
20

Description

Is your feature request related to a problem?

Currently the only way for customers to have more variation on the current templates we offer is through adding another template, or through substituting the given defaults on the domain:

POST /_plugins/_flow_framework/workflow?use_case=semantic_search_with_cohere_embedding
{
    "create_connector.model" : "embed-multilingual-v3.0",
    "text_embedding.field_map.output": "book_embedding",
    "create_index.name": "sparse-book-index"
}

Some users might want to have the ability to give a lot more settings for the create index step of a template or add 2 more unique nested field on their index to test their hybrid queries.

example substitution ready create index step:

{
            "index_name": "${{create_index.name}}",
            "configurations": {
              "settings": {
                "index.knn": true,
                "default_pipeline": "${{create_ingest_pipeline.pipeline_id}}",
                "number_of_shards": "${{create_index.settings.number_of_shards}}",
                "index.search.default_pipeline": "${{create_search_pipeline.pipeline_id}}"
              },
              "mappings": {
                "_doc": {
                  "properties": {
                    "id": {
                      "type": "text"
                    },
                    "${{text_embedding.field_map.output}}": {
                      "type": "knn_vector",
                      "dimension": "${{text_embedding.field_map.output.dimension}}",
                      "method": {
                        "engine": "${{create_index.mappings.method.engine}}",
                        "space_type": "${{create_index.mappings.method.space_type}}",
                        "name": "${{create_index.mappings.method.name}}",
                        "parameters": {}
                      }
                    },
                    "${{text_embedding.field_map.input}}": {
                      "type": "text"
                    }
                  }
                }
              }
            }

users might want to add one more field to this index for example ^

Additionally users have already asked if they can use the hybrid_search use case but without any weights at all. However we currently don't offer substitution for the entire field if its already in the pre-defined template. Users can only overwrite the actual value of the weights field.

What solution would you like?

We should think of a mechanism where it is to substitute in entire steps of a template or for example have additional pre-defined templates where if user's don't specify any substitution we proceed with the template we have today but if they give us their entire index mapping we can just substitute that entire body.

For the request on not having weights at all, a simple solution could be to have two templates for hybrid_search on the backend but only expose one use case to users hybrid_search and if the users don't define any weights in API request then we will now use the template with no weights and vice versa.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the POST /_plugins/_flow_framework/workflow endpoint and trace how substitutions are applied to default use-case templates, including semantic_search_with_cohere_embedding and hybrid_search. Define what is required for replacing whole template steps or mappings and selecting a no-weights hybrid template; the issue does not name implementation files or tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend-api-design
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.