Are the EMR-related constants working?
- Dominant language
- Python
- Stars
- 7.3k
- Forks
- 1.4k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 15
Description
## Expected Behavior
Expected behavior when the following environment variables are set:
```
FEAST_SPARK_STAGING_LOCATION=s3://...
FEAST_SPARK_LAUNCHER="emr"
FEAST_EMR_CLUSTER_ID=""
FEAST_EMR_LOG_LOCATION=""
```
1. Successfully connect to the specified EMR cluster.
2. Initialize a SparkSession within the EMR environment.
3. Execute the defined Spark job on the cluster.
4. Upon completion of the Spark job, terminate the SparkSession and release cluster resources.
5. Store the results of the Spark job in the online store.
## Current Behavior
No Spark job is submitted to the EMR cluster. The code exits silently without any indication of success or failure of the materialization.
## Steps to reproduce
```
FEAST_SPARK_STAGING_LOCATION=s3://...
FEAST_SPARK_LAUNCHER="emr"
FEAST_EMR_CLUSTER_ID=""
FEAST_EMR_LOG_LOCATION=""
```
RepoConfig:
```
{
"project": "",
"registry": "s3://",
"provider": "aws",
"entity_key_serialization_version": 2,
"online_store":
{
"type": "dynamodb",
"region": "us-west-2"
},
"offline_store":
{
"type": "spark",
"region": "us-west-2",
"staging_location": "s3://",
"spark_conf":
{
"spark.master":"",
"spark.ui.enabled": "true",
"spark.eventLog.enabled": "false",
"spark.sql.catalogImplementation": "hive",
"spark.sql.parser.quotedRegexColumnNames": "true",
"spark.sql.session.timeZone": "UTC",
"spark.jars.packages": "org.apache.hadoop:hadoop-aws:3.3.1"
}
},
"batch_engine":
{
"type": "spark.engine"
}
}
```
### Specifications
- Version:
`feast = { extras = ["aws", "gcp", "spark"], version = "==0.42.0" }`
- Platform: macOS
## Possible Solution
The only code reference found for the constant `FEAST_SPARK_LAUNCHER` dates back four years and is located here: https://github.com/feast-dev/feast/blob/dc2c1dc67c8e191c29155ced3334244351f312c7/infra/terraform/aws/helm.tf#L93
Are the EMR-related constants working in the current FEAST version (0.42.0)?
```
# EMR cluster to run Feast Spark Jobs in
EMR_CLUSTER_ID: Optional[str] = None
# Region of EMR cluster
EMR_REGION: Optional[str] = None
# Template path of EMR cluster
EMR_CLUSTER_TEMPLATE_PATH: Optional[str] = None
# Log path of EMR cluster
EMR_LOG_LOCATION: Optional[str] = None
```
References:
- https://feast-spark.readthedocs.io/en/stable/_modules/feast_spark/constants.html
- https://docs.feast.dev/v0.11-branch/feast-on-kubernetes/reference-1/feast-and-spark#option-3.-use-aws-and-emr
Contributor guide
Assessment
This issue has not been assessed yet.