feast-dev / feast-dev/feast

Are the EMR-related constants working?

Open
#4,968 1 comment 1 reaction 0 assignees View on GitHub
kind/bug priority/p2
Dominant language
Python
Stars
7.3k
Forks
1.4k
Avg merge
1d 21h
Merged PRs (30d)
15

Description

## Expected Behavior

Expected behavior when the following environment variables are set:

```
FEAST_SPARK_STAGING_LOCATION=s3://...
FEAST_SPARK_LAUNCHER="emr"
FEAST_EMR_CLUSTER_ID=""
FEAST_EMR_LOG_LOCATION=""
```

1. Successfully connect to the specified EMR cluster.
2. Initialize a SparkSession within the EMR environment.
3. Execute the defined Spark job on the cluster.
4. Upon completion of the Spark job, terminate the SparkSession and release cluster resources.
5. Store the results of the Spark job in the online store.

## Current Behavior

No Spark job is submitted to the EMR cluster. The code exits silently without any indication of success or failure of the materialization.

## Steps to reproduce

```
FEAST_SPARK_STAGING_LOCATION=s3://...
FEAST_SPARK_LAUNCHER="emr"
FEAST_EMR_CLUSTER_ID=""
FEAST_EMR_LOG_LOCATION=""
```

RepoConfig:

```
{
"project": "",
"registry": "s3://",
"provider": "aws",
"entity_key_serialization_version": 2,
"online_store":
{
"type": "dynamodb",
"region": "us-west-2"
},
"offline_store":
{
"type": "spark",
"region": "us-west-2",
"staging_location": "s3://",
"spark_conf":
{
"spark.master":"",
"spark.ui.enabled": "true",
"spark.eventLog.enabled": "false",
"spark.sql.catalogImplementation": "hive",
"spark.sql.parser.quotedRegexColumnNames": "true",
"spark.sql.session.timeZone": "UTC",
"spark.jars.packages": "org.apache.hadoop:hadoop-aws:3.3.1"
}
},
"batch_engine":
{
"type": "spark.engine"
}
}
```

### Specifications

- Version:
`feast = { extras = ["aws", "gcp", "spark"], version = "==0.42.0" }`
- Platform: macOS

## Possible Solution

The only code reference found for the constant `FEAST_SPARK_LAUNCHER` dates back four years and is located here: https://github.com/feast-dev/feast/blob/dc2c1dc67c8e191c29155ced3334244351f312c7/infra/terraform/aws/helm.tf#L93

Are the EMR-related constants working in the current FEAST version (0.42.0)?

```
# EMR cluster to run Feast Spark Jobs in
EMR_CLUSTER_ID: Optional[str] = None

# Region of EMR cluster
EMR_REGION: Optional[str] = None

# Template path of EMR cluster
EMR_CLUSTER_TEMPLATE_PATH: Optional[str] = None

# Log path of EMR cluster
EMR_LOG_LOCATION: Optional[str] = None
```

References:
- https://feast-spark.readthedocs.io/en/stable/_modules/feast_spark/constants.html
- https://docs.feast.dev/v0.11-branch/feast-on-kubernetes/reference-1/feast-and-spark#option-3.-use-aws-and-emr

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.