microsoft / microsoft/SynapseML
[BUG] Unable to install synapse ml on AWS Glue to use LightGBM library
- Dominant language
- Scala
- Stars
- 5.2k
- Forks
- 868
- Avg merge
- 22h 9m
- Merged PRs (30d)
- 45
Description
### SynapseML version
0.11.2
### System information
- **Language version** (e.g. python 3.8, scala 2.12): Python 3.8
- **Spark Version** (e.g. 3.2.3): 3.3
- **Spark Platform** (e.g. Synapse, Databricks): AWS Glue
### Describe the problem
I am trying to install Synapse ML on AWS Glue to use LightGBM library and getting this error
ModuleNotFoundError: No module named 'synapse'
### Code to reproduce issue
import pyspark
from awsglue.context import GlueContext
from awsglue.job import Job
import pyspark.sql.functions as f
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext, SparkConf
conf = SparkConf().setAppName("SynapseTest").set("spark.sql.sources.partitionOverwriteMode", "dynamic").set("spark.jars.packages", "com.microsoft.azure/synapseml_2.12:0.11.2").set("spark.jars.repositories", "https://repo1.maven.org/maven2/")
sc = SparkContext(conf=conf)
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
args = getResolvedOptions(sys.argv, ['JOB_NAME'])
print("args", args)
job.init(args['JOB_NAME'], args)
import synapse.ml
### Other info / logs
I have also tried --extra-jars argument and given jar file location from s3 bucket. Still didn't work
### What component(s) does this bug affect?
- [ ] `area/cognitive`: Cognitive project
- [X] `area/core`: Core project
- [ ] `area/deep-learning`: DeepLearning project
- [ ] `area/lightgbm`: Lightgbm project
- [ ] `area/opencv`: Opencv project
- [ ] `area/vw`: VW project
- [ ] `area/website`: Website
- [ ] `area/build`: Project build system
- [ ] `area/notebooks`: Samples under notebooks folder
- [ ] `area/docker`: Docker usage
- [ ] `area/models`: models related issue
### What language(s) does this bug affect?
- [ ] `language/scala`: Scala source code
- [X] `language/python`: Pyspark APIs
- [ ] `language/r`: R APIs
- [ ] `language/csharp`: .NET APIs
- [ ] `language/new`: Proposals for new client languages
### What integration(s) does this bug affect?
- [X] `integrations/synapse`: Azure Synapse integrations
- [ ] `integrations/azureml`: Azure ML integrations
- [ ] `integrations/databricks`: Databricks integrations
Contributor guide
Research direction
Start with the reproduction configuration using Python 3.8, Spark 3.3, AWS Glue, and SynapseML 0.11.2, then compare the Maven package and --extra-jars approaches described in the issue. Confirm whether the AWS Glue job can import synapse.ml and use the LightGBM library; the reported ModuleNotFoundError is the failure to resolve.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cloud, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100