opensearch-project / opensearch-project/OpenSearch

Support different compression types for snapshoting

Open
#2,192 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

discuss distributed framework enhancement
Dominant language
Java
Stars
13.7k
Forks
3k
Avg merge
2d 23h
Merged PRs (30d)
108

Description

Is your feature request related to a problem? Please describe.

For now Opensearch supports only deflate compression for snapshoting (compress setting for all Cloud plugins).
It would be great to extend compression to use different compression algorithms like:

  • ZSTD
  • LZ4
    or even more.

Describe the solution you'd like
To support different compression types by
extending the repository settings for compression using such JSON:

{
  ...
   "settings": {
      "compression": {
         "type": "zstd", // could be `deflate`, `xz4` etc.
         "compression_settings": {
              // different compression settings like level and so on
         }
      }
   }
}

In this case it is difficult to keep backward compatibility

Describe alternatives you've considered
Compression type could be set when the repository is created/updated using such JSON:

{
  ...
   "settings": {
      "compress": true,
      "compression_type": "zstd", // `deflate`, `lz4`
   }
}

and add compression settings in opensearch.yml something like:

compression.<propery>: <some_property_value>

In this case it is possible to keep backward compatibility between versions. e.g. deflate is a default type for compression

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the snapshot repository settings and the existing deflate handling in the OpenSearch codebase, then review how Cloud plugins and opensearch.yml currently configure compression. Compare the proposed repository-level and configuration-level approaches, including backward compatibility. Done means supported compression types and settings are defined with a compatible default and covered by the relevant tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.