opensearch-project / opensearch-project/sql

[FEATURE] Migrate index schema fetch api from _mappings to _field_caps

Open
#4,551 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement PPL
Dominant language
Java
Stars
176
Forks
229
Avg merge
2d 21h
Merged PRs (30d)
43

Description

Is your feature request related to a problem?
Since OpenSearch does not support cross cluster index metadata retrieval, field mapping of a remote cluster index is not available to the local cluster. (https://github.com/opensearch-project/OpenSearch/issues/6573) Therefore, the query engine requires that for any remote cluster index that the users need to search, the local cluster keep a field mapping system index with the same index name. This can be done by creating an index on the local cluster with the same name and schema as the remote cluster index.

https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/admin/cross_cluster_search.rst#limitations

What solution would you like?
User can query index in remote cluster without setup index in local cluster.

###
POST {{local}}/_plugins/_ppl
Content-Type: application/x-ndjson

{
    "query": "source=`remoteCluster:products` | stats count() by id"
}
  • _mappings API does not support refering to remote cluster index
###
GET {{local}}/remoteCluster:products/_mappings

### Response
{
  "error": {
    "root_cause": [
      {
        "type": "index_not_found_exception",
        "reason": "no such index [b:products]",
        "index": "b:products",
        "resource.id": "b:products",
        "resource.type": "index_or_alias",
        "index_uuid": "_na_"
      }
    ],
    "type": "index_not_found_exception",
    "reason": "no such index [b:products]",
    "index": "b:products",
    "resource.id": "b:products",
    "resource.type": "index_or_alias",
    "index_uuid": "_na_"
  },
  "status": 404
}
  • _field_caps can support refer remote cluster index in local cluster
###
GET {{local}}/remoteCluster:products/_field_caps?fields=*

### Response
{
  "indices": [
    "b:products"
  ],
  "fields": {
    "price": {
      "double": {
        "type": "double",
        "searchable": true,
        "aggregatable": true
      }
    },
    "name": {
      "keyword": {
        "type": "keyword",
        "searchable": true,
        "aggregatable": true
      }
    },
    "id": {
      "keyword": {
        "type": "keyword",
        "searchable": true,
        "aggregatable": true
      }
    },
    "category": {
      "keyword": {
        "type": "keyword",
        "searchable": true,
        "aggregatable": true
      }
    },

  }
}

What alternatives have you considered?
n/a

Do you have any additional context?
Follow READE.me to setup cross cluster, use ccs.http to test PPL cross cluster.
https://github.com/penghuo/BusyBox/tree/main/ppl/ccs

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with docs/user/ppl/admin/cross_cluster_search.rst and the linked BusyBox ccs.http example, then trace the schema request that currently uses the _mappings endpoint. Compare it with the documented _field_caps response and test the PPL query against a remote products index; done when it works without a same-named local index.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.