opensearch-project / opensearch-project/sql
[FEATURE] Add abstraction to Async API for using Spark
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 176
- Forks
- 229
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 43
Description
Currently the async API is designed to use the AWS EMR service when it needs Spark to process a query. For example, the EMRServerlessClientFactory class is used for getting an instance of AWSEMRServerless.
There should be an abstraction here so that async API uses an abstract service to start and manage Spark jobs. EMR would be an implementation of this abstract service. Another possible implementation is to use Docker.
Is your feature request related to a problem?
No
What solution would you like?
An abstract service for managing Spark jobs, along with an EMR implementation of this service.
What alternatives have you considered?
None, open to discussion.
Do you have any additional context?
In this OpenSearch Spark PR, the aws-java-sdk-emrserverless Jar file is updated to replace the AWSEMRServerless implementation. There should be a cleaner way of replacing usage of the EMR service.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading AsyncExecutorServiceModule.java around line 250 and EMRServerlessClientFactoryImpl.java around line 62, then review the linked OpenSearch Spark PR. Identify the abstraction needed for starting and managing Spark jobs, with EMR as its implementation. Done means the async API no longer depends directly on the EMR service and the EMR implementation remains usable.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, java
- Domain
- api, backend, cloud
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100