opensearch-project / opensearch-project/data-prepper

Ingest data from ODBC/JDBC datasources as Source

Open
#1,995 1 comment 4 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Java
Stars
374
Forks
354
Avg merge
3d 18h
Merged PRs (30d)
8

Description

Is your feature request related to a problem? Please describe.

Pipeline author wants to read data from desired databases. Plugin needs to provide interface to read/ingest data with a JDBC interface.

Describe the solution you'd like

The interface should also provide ability to read data and run queries periodically. Every row read will be converted to Data prepper event. Columns should be mapped as fields in event. I would envision JDBC driver libraries to be provided in yml configuration by pipeline author. User should be able to pass required configuration for drivers under "jdbc_driver_lib". Additionally for scheduling a periodic run a cron like syntax configuration should be passed in the yml.

The plugin should be able to support SigV4 and accept awsCrediential provider, region, security parameters in yml configuration which should include trust & keystore configurations - trustStoreLocation,trustStoreType,trustStorePassword, keyStoreLocation,KeyStoreType,keyStorePassword

The plugin should include support for multi-node worker partitioning.

source:
    - jdbc:
          jdbc_driver_lib: "jdbc-oracle.jar"
          jbdc_driver:"oracle.jdbc.driver.OracleDriver"
          jdbc_connection_string:"jdbc:oracle://127.0.0.1:8080"
          jdbc_user:"user"
          jdbc_schedule:"* * * 3 *"
          sql_query:"SELECT EMPLOYEE_ID FROM EMPLOYEES WHERE LAST_NAME= :LAST_NAME"
          fetchSize: " "
          awsCredentialsProvider: "com.amazonaws.opensearch.sql.jdbc.shadow.com.amazonaws.auth.AWSCredentialsProvider"
        

Additional context

https://github.com/opensearch-project/sql-jdbc

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue names no repository files, tests, or existing entry points. Start by reviewing the linked OpenSearch SQL JDBC project and the Data Prepper source-plugin conventions, then define the JDBC configuration, scheduling, credentials, TLS, event mapping, and worker-partitioning requirements before implementation. Done means the feature supports the requested ingestion scenarios with documented configuration and coverage for its behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
backend, data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
15/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.