microsoft / microsoft/SynapseML

Error: parameter numItemBlocks given invalid value 0

Open
#1,346 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area/cognitive-service bug
Dominant language
Scala
Stars
5.2k
Forks
868
Avg merge
22h 9m
Merged PRs (30d)
45

Description

**Describe the bug**
When running the following lines, I get the error: parameter numItemBlocks given invalid value 0.

access_anomaly = AccessAnomaly(
tenantCol='tenant_id',
userCol='user',
resCol='res',
likelihoodCol='likelihood',
maxIter=100
)
access_anomaly_model = access_anomaly.fit(ptraining)

ptraining was set to a dataset created from sample data

**To Reproduce**
Running the code in the AnomalousRATraining notebook that is part of the MS Sentinel Bring Your Own ML sample

**Info (please complete the following information):**
- SynapseML Version: unknown
- Spark Version: 3.1.2
- Spark Platform : Azure Databricks

** Stacktrace**
```
Please post the stacktrace here if applicable---------------------------------------------------------------------------

IllegalArgumentException Traceback (most recent call last)

in

7 maxIter=100

8 )

----> 9 access_anomaly_model = access_anomaly.fit(ptraining)

/databricks/python_shell/dbruntime/MLWorkloadsInstrumentation/_pyspark.py in patched_method(self, *args, **kwargs)

28 call_succeeded = False

29 try:

---> 30 result = original_method(self, *args, **kwargs)

31 call_succeeded = True

32 return result

/databricks/spark/python/pyspark/ml/base.py in fit(self, dataset, params)

159 return self.copy(params)._fit(dataset)

160 else:

--> 161 return self._fit(dataset)

162 else:

163 raise ValueError("Params must be either a param map or a list/tuple of param maps, "

/local_disk0/spark-e510fb9f-d7e4-4c84-95a6-a0d6ff9719e5/userFiles-f183d53d-3671-46c7-aaca-03a6002001e0/addedFile2895442502225420743synapseml_core_2_12_0_9_5-06d81.jar/synapse/ml/cyber/anomaly/collaborative_filtering.py in _fit(self, df)

837 enriched_df = self._enrich_and_normalize(indexed_df).cache()

838

--> 839 user_res_feature_vector_mapping_df = self.create_spark_model_vectors_df(enriched_df)

840 user_res_norm_cf_df_model = ModelNormalizeTransformer(

841 enriched_df, self.rank_param

/local_disk0/spark-e510fb9f-d7e4-4c84-95a6-a0d6ff9719e5/userFiles-f183d53d-3671-46c7-aaca-03a6002001e0/addedFile2895442502225420743synapseml_core_2_12_0_9_5-06d81.jar/synapse/ml/cyber/anomaly/collaborative_filtering.py in create_spark_model_vectors_df(self, df)

795 ) if res_mapping_df is not None else curr_res_mapping_df

796 else:

--> 797 user_mapping_df, res_mapping_df = self._train_cf(als, df)

798

799 assert user_mapping_df is not None and res_mapping_df is not None

/local_disk0/spark-e510fb9f-d7e4-4c84-95a6-a0d6ff9719e5/userFiles-f183d53d-3671-46c7-aaca-03a6002001e0/addedFile2895442502225420743synapseml_core_2_12_0_9_5-06d81.jar/synapse/ml/cyber/anomaly/collaborative_filtering.py in _train_cf(self, als, df)

724 res_vec_col = self.res_vec_col

725

--> 726 spark_model = als.fit(df)

727

728 user_mapping_df = spark_model.userFactors.select(

/databricks/python_shell/dbruntime/MLWorkloadsInstrumentation/_pyspark.py in patched_method(self, *args, **kwargs)

28 call_succeeded = False

29 try:

---> 30 result = original_method(self, *args, **kwargs)

31 call_succeeded = True

32 return result

/databricks/spark/python/pyspark/ml/base.py in fit(self, dataset, params)

159 return self.copy(params)._fit(dataset)

160 else:

--> 161 return self._fit(dataset)

162 else:

163 raise ValueError("Params must be either a param map or a list/tuple of param maps, "

/databricks/spark/python/pyspark/ml/wrapper.py in _fit(self, dataset)

333

334 def _fit(self, dataset):

--> 335 java_model = self._fit_java(dataset)

336 model = self._create_model(java_model)

337 return self._copyValues(model)

/databricks/spark/python/pyspark/ml/wrapper.py in _fit_java(self, dataset)

329 fitted Java model

330 """

--> 331 self._transfer_params_to_java()

332 return self._java_obj.fit(dataset._jdf)

333

/databricks/spark/python/pyspark/ml/wrapper.py in _transfer_params_to_java(self)

141 for param in self.params:

142 if self.isSet(param):

--> 143 pair = self._make_java_param_pair(param, self._paramMap[param])

144 self._java_obj.set(pair)

145 if self.hasDefault(param):

/databricks/spark/python/pyspark/ml/wrapper.py in _make_java_param_pair(self, param, value)

132 java_param = self._java_obj.getParam(param.name)

133 java_value = _py2java(sc, value)

--> 134 return java_param.w(java_value)

135

136 def _transfer_params_to_java(self):

/databricks/spark/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py in __call__(self, *args)

1302

1303 answer = self.gateway_client.send_command(command)

-> 1304 return_value = get_return_value(

1305 answer, self.gateway_client, self.target_id, self.name)

1306

/databricks/spark/python/pyspark/sql/utils.py in deco(*a, **kw)

121 # Hide where the exception came from that shows a non-Pythonic

122 # JVM exception message.

--> 123 raise converted from None

124 else:

125 raise

IllegalArgumentException: ALS_93abf3745fbf parameter numItemBlocks given invalid value 0.
```

AB#1884828

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the failure in the AnomalousRATraining notebook, then start with synapse/ml/cyber/anomaly/collaborative_filtering.py and the _train_cf and create_spark_model_vectors_df paths shown in the stack trace. Check how the ALS fit receives numItemBlocks for the sample dataset and add a regression test or validation that demonstrates the corrected behavior. Done means the sample training flow no longer fails with an invalid zero block count.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, machine-learning, python, scala
Domain
data-engineering, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.