opensearch-project / opensearch-project/data-prepper

[BUG] Cannot include hidden indices

Open
#7,026 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement follow up question
Dominant language
Java
Stars
374
Forks
354
Avg merge
3d 18h
Merged PRs (30d)
8

Description

Describe the bug
Indices are searched using /_cat/indices?format=json, which means hidden indices can never be included.

To Reproduce

Create an hidden index in OpenSearch:

PUT /my-hidden-index
{
  "settings": {
    "index.hidden": "true"
  }
}

Write any pipeline that tries to include that index:

# ...
indices:
  include:
    - index_name_regex: "my-hidden-index"
# ...

See the logs saying that no indices matched:

DEBUG org.opensearch.client.RequestLogger - request [GET http://opensearch:9200/_cat/indices?format=json] returned [HTTP/1.1 200 OK]
DEBUG org.opensearch.dataprepper.plugins.source.opensearch.worker.OpenSearchIndexPartitionCreationSupplier - Found 9 indices
WARN  org.opensearch.dataprepper.plugins.source.opensearch.worker.OpenSearchIndexPartitionCreationSupplier - No indices matched the configured regex patterns after applying include/exclude filters

Expected behavior
It should be able to include hidden indices just like it can already include system-indices.

For backwards compatibility, this could be an opt-in flag (e.g., include_hidden: true).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with OpenSearchIndexPartitionCreationSupplier and trace the request to /_cat/indices?format=json. Run the supplied hidden-index and pipeline reproduction, then verify that the configured hidden index can be matched while existing behavior remains compatible, including any opt-in setting selected by the maintainers.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, search
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.