opensearch-project / opensearch-project/sql

[FEATURE] Add abstraction to Async API for using Spark

Open
#3,252 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

async enhancement
Dominant language
Java
Stars
176
Forks
229
Avg merge
2d 21h
Merged PRs (30d)
43

Description

Currently the async API is designed to use the AWS EMR service when it needs Spark to process a query. For example, the EMRServerlessClientFactory class is used for getting an instance of AWSEMRServerless.

https://github.com/opensearch-project/sql/blob/main/async-query/src/main/java/org/opensearch/sql/spark/transport/config/AsyncExecutorServiceModule.java#L250

https://github.com/opensearch-project/sql/blob/main/async-query-core/src/main/java/org/opensearch/sql/spark/client/EMRServerlessClientFactoryImpl.java#L62

There should be an abstraction here so that async API uses an abstract service to start and manage Spark jobs. EMR would be an implementation of this abstract service. Another possible implementation is to use Docker.

Is your feature request related to a problem?
No

What solution would you like?
An abstract service for managing Spark jobs, along with an EMR implementation of this service.

What alternatives have you considered?
None, open to discussion.

Do you have any additional context?
In this OpenSearch Spark PR, the aws-java-sdk-emrserverless Jar file is updated to replace the AWSEMRServerless implementation. There should be a cleaner way of replacing usage of the EMR service.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading AsyncExecutorServiceModule.java around line 250 and EMRServerlessClientFactoryImpl.java around line 62, then review the linked OpenSearch Spark PR. Identify the abstraction needed for starting and managing Spark jobs, with EMR as its implementation. Done means the async API no longer depends directly on the EMR service and the EMR implementation remains usable.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java
Domain
api, backend, cloud
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.