Value modelsRDD in class RandomEffectModel cannot be accessed
Nobody has claimed this yet.
- Dominant language
- Terra
- Stars
- 795
- Forks
- 175
- PR merge metrics
- No merged PRs in 30d
Description
Hi
I'm trying to replicate code in lme4-photon-sleep.zip in https://github.com/linkedin/photon-ml/issues/374 using Apache Toree notebook (directly accessing PhotonML API) Spark version 2.3.4 with scala 2.11.8 and latest Photon version jar.
I get error when trying to save random effects from model in the following step
```
val dx_re = mixedModel.toMap.get("rand").get match {
case m: RandomEffectModel =>
val reModelRDD = m.modelsRDD
val re_tup = reModelRDD.map(x => (x._1, x._2.coefficients.means.toArray.zipWithIndex))
val re_flat = re_tup.flatMap{
case (store, arr) => {
arr.map(cell => (store, cell))
}
}
val coeffNames = rand_effects.columns.slice(1,rf_cols+2)
val re_flat_col = re_flat.map(store_tup => (store_tup._1, coeffNames.apply(store_tup._2._2), store_tup._2._1))
val dx_re = spark.createDataFrame(re_flat_col).selectExpr("_1 as store", "_2 as column", "_3 as coeff")
dx_re
}
dx_re.show()
```
> Message: :82: error: value modelsRDD in class RandomEffectModel cannot be accessed in com.linkedin.photon.ml.model.RandomEffectModel
> Access to protected value modelsRDD not permitted because
> enclosing class $iw is not a subclass of
> class RandomEffectModel in package model where target is defined
> val reModelRDD = m.modelsRDD
Is this something related to my environment or changes in Photon ML API?
If related to Photon ML how can I see/save random effects without using ModelProcessingUtils.saveGameModelToHDFS?
Problem with ModelProcessingUtils.saveGameModelToHDFS is that it requires inputIndexMaps that is generated by AvroReader that I'd like to avoid as my data is in csv.
thanks
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by examining RandomEffectModel.modelsRDD and the access error, then trace ModelProcessingUtils.saveGameModelToHDFS to understand how random effects are exposed. Compare the documented API with the lme4-photon-sleep.zip example and the CSV use case; done means identifying a supported way to inspect or save random effects without Avro-generated inputIndexMaps.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- spark
- Domain
- api, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100