opensearch-project / opensearch-project/data-prepper

[FEATURE] Store the source partition key as decomposed columns in the JDBC source coordination store

Open
#6,982 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Java
Stars
374
Forks
355
Avg merge
3d 18h
Merged PRs (30d)
8

Description

Is your feature request related to a problem? Please describe.
In the JDBC source coordination store (#6757), source_partition_key is a single opaque VARCHAR(512) column that forms the composite primary key together with source_identifier. Source plugins encode structured information into this key (for example bucket, object key, and scan range for S3-based sources), but the store flattens it into one string.

As a result:

  • Operational queries against the coordination table (e.g. "show all partitions for bucket X") require LIKE matching on the encoded string
  • The whole key must fit in one column, which is capped at VARCHAR(512) by MySQL's 3072-byte composite index limit
  • Individual key components cannot be indexed

Describe the solution you'd like
Store the partition key components in separate columns in the JDBC store schema, keeping (source_identifier, source_partition_key) as the logical identity. For example, nullable component columns (partition_key_part_1 .. partition_key_part_n), or a JSON column on databases that support it.

Design considerations
Today each source plugin invents its own key format, and the SourceCoordinationStore interface in data-prepper-api only sees an opaque string: S3-based sources use bucket|key-style segments, DynamoDB uses export ARNs and shard IDs, Iceberg uses file paths and ranges. There is no cross-source convention, not even for the separator character.

So decomposing the key requires one of:

  1. A documented key convention that all source plugins follow (a cross-plugin contract, and existing keys in the wild would not conform), or
  2. An extension of the SourceCoordinationStore interface so sources pass structured components explicitly (an API change affecting every store implementation, including DynamoDB and in-memory)

Additional context
The single-column schema shipped in #6757 keeps working as is. Adding nullable component columns later is backward compatible, so deferring this does not create a migration problem.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the SourceCoordinationStore interface in data-prepper-api and the JDBC coordination-store schema introduced in #6757. Compare how the DynamoDB and in-memory implementations represent keys, then document the chosen cross-source contract or API change, including compatibility for existing keys. Done means the design is agreed and all affected store implementations have a clear migration path.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
backend-api-design, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.