opensearch-project / opensearch-project/sql
[FEATURE] Migrate index schema fetch api from _mappings to _field_caps
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 176
- Forks
- 229
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 43
Description
Is your feature request related to a problem?
Since OpenSearch does not support cross cluster index metadata retrieval, field mapping of a remote cluster index is not available to the local cluster. (https://github.com/opensearch-project/OpenSearch/issues/6573) Therefore, the query engine requires that for any remote cluster index that the users need to search, the local cluster keep a field mapping system index with the same index name. This can be done by creating an index on the local cluster with the same name and schema as the remote cluster index.
What solution would you like?
User can query index in remote cluster without setup index in local cluster.
###
POST {{local}}/_plugins/_ppl
Content-Type: application/x-ndjson
{
"query": "source=`remoteCluster:products` | stats count() by id"
}
- _mappings API does not support refering to remote cluster index
###
GET {{local}}/remoteCluster:products/_mappings
### Response
{
"error": {
"root_cause": [
{
"type": "index_not_found_exception",
"reason": "no such index [b:products]",
"index": "b:products",
"resource.id": "b:products",
"resource.type": "index_or_alias",
"index_uuid": "_na_"
}
],
"type": "index_not_found_exception",
"reason": "no such index [b:products]",
"index": "b:products",
"resource.id": "b:products",
"resource.type": "index_or_alias",
"index_uuid": "_na_"
},
"status": 404
}
- _field_caps can support refer remote cluster index in local cluster
###
GET {{local}}/remoteCluster:products/_field_caps?fields=*
### Response
{
"indices": [
"b:products"
],
"fields": {
"price": {
"double": {
"type": "double",
"searchable": true,
"aggregatable": true
}
},
"name": {
"keyword": {
"type": "keyword",
"searchable": true,
"aggregatable": true
}
},
"id": {
"keyword": {
"type": "keyword",
"searchable": true,
"aggregatable": true
}
},
"category": {
"keyword": {
"type": "keyword",
"searchable": true,
"aggregatable": true
}
},
}
}
What alternatives have you considered?
n/a
Do you have any additional context?
Follow READE.me to setup cross cluster, use ccs.http to test PPL cross cluster.
https://github.com/penghuo/BusyBox/tree/main/ppl/ccs
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with docs/user/ppl/admin/cross_cluster_search.rst and the linked BusyBox ccs.http example, then trace the schema request that currently uses the _mappings endpoint. Compare it with the documented _field_caps response and test the PPL query against a remote products index; done when it works without a same-named local index.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100