microsoft / microsoft/SynapseML

[BUG] Unable to install synapse ml on AWS Glue to use LightGBM library

Open
#2,055 2 comments 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
5.2k
Forks
868
Avg merge
22h 9m
Merged PRs (30d)
45

Description

### SynapseML version

0.11.2

### System information

- **Language version** (e.g. python 3.8, scala 2.12): Python 3.8
- **Spark Version** (e.g. 3.2.3): 3.3
- **Spark Platform** (e.g. Synapse, Databricks): AWS Glue

### Describe the problem

I am trying to install Synapse ML on AWS Glue to use LightGBM library and getting this error

ModuleNotFoundError: No module named 'synapse'

### Code to reproduce issue

import pyspark
from awsglue.context import GlueContext
from awsglue.job import Job
import pyspark.sql.functions as f
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext, SparkConf

conf = SparkConf().setAppName("SynapseTest").set("spark.sql.sources.partitionOverwriteMode", "dynamic").set("spark.jars.packages", "com.microsoft.azure/synapseml_2.12:0.11.2").set("spark.jars.repositories", "https://repo1.maven.org/maven2/")

sc = SparkContext(conf=conf)
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
args = getResolvedOptions(sys.argv, ['JOB_NAME'])
print("args", args)
job.init(args['JOB_NAME'], args)
import synapse.ml

### Other info / logs

I have also tried --extra-jars argument and given jar file location from s3 bucket. Still didn't work

### What component(s) does this bug affect?

- [ ] `area/cognitive`: Cognitive project
- [X] `area/core`: Core project
- [ ] `area/deep-learning`: DeepLearning project
- [ ] `area/lightgbm`: Lightgbm project
- [ ] `area/opencv`: Opencv project
- [ ] `area/vw`: VW project
- [ ] `area/website`: Website
- [ ] `area/build`: Project build system
- [ ] `area/notebooks`: Samples under notebooks folder
- [ ] `area/docker`: Docker usage
- [ ] `area/models`: models related issue

### What language(s) does this bug affect?

- [ ] `language/scala`: Scala source code
- [X] `language/python`: Pyspark APIs
- [ ] `language/r`: R APIs
- [ ] `language/csharp`: .NET APIs
- [ ] `language/new`: Proposals for new client languages

### What integration(s) does this bug affect?

- [X] `integrations/synapse`: Azure Synapse integrations
- [ ] `integrations/azureml`: Azure ML integrations
- [ ] `integrations/databricks`: Databricks integrations

Contributor guide

Open the contributing guide

Research direction

Start with the reproduction configuration using Python 3.8, Spark 3.3, AWS Glue, and SynapseML 0.11.2, then compare the Maven package and --extra-jars approaches described in the issue. Confirm whether the AWS Glue job can import synapse.ml and use the LightGBM library; the reported ModuleNotFoundError is the failure to resolve.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.