deepjavalibrary / deepjavalibrary/djl

Getting java.lang.IllegalStateException: Tried to specify the thread pool when creating an OrtEnvironment, but one already exists.

Open
#3,679 10 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
4.9k
Forks
759
Avg merge
16h 15m
Merged PRs (30d)
15

Description

We are seeing a
`Getting java.lang.IllegalStateException: Tried to specify the thread pool when creating an OrtEnvironment, but one already exists.`

This is our object to create the criteria

`object ModelCriteria {
def createCriteria(runTime: String,
modelFilePath: String,
modelFileName: String,
translator: Translator[Map[String, Any], Map[String, Any]]
): Criteria[Map[String, Any], Map[String, Any]] = {

// Manually setting the number of threads to be used by the Pytorch Engine
System.setProperty("ai.djl.pytorch.num_threads", sys.env.getOrElse("PYTORCH_NUM_THREADS", "1"))
System.setProperty("ai.djl.pytorch.num_interop_threads", sys.env.getOrElse("PYTORCH_NUM_INTEROP_THREADS", "1"))

// Using interOpNumThreads and intraOpNumThreads option in Criteria builder to control parallelism
val criteria: Criteria[Map[String, Any], Map[String, Any]] = runTime match {
case "OnnxRuntime" =>
Criteria.builder()
.optOption("interOpNumThreads", sys.env.getOrElse("ONNX_INTEROP_NUM_THREADS", "1"))
.optOption("intraOpNumThreads", sys.env.getOrElse("ONNX_INTRAOP_NUM_THREADS", "1"))
.setTypes(classOf[Map[String, Any]], classOf[Map[String, Any]])
.optTranslator(translator)
.optModelUrls(modelFilePath)
.optModelName(modelFileName)
.optEngine(runTime)
.optOption("mapLocation", "true")
.optProgress(new ProgressBar())
.build()

case _ =>
Criteria.builder()
.setTypes(classOf[Map[String, Any]], classOf[Map[String, Any]])
.optTranslator(translator)
.optModelUrls(modelFilePath)
.optModelName(modelFileName)
.optEngine(runTime)
.optOption("mapLocation", "true")
.optProgress(new ProgressBar())
.build()

}
criteria
}
}`

When we try to load two onnx models back to back we are seeing this issue

`Getting java.lang.IllegalStateException: Tried to specify the thread pool when creating an OrtEnvironment, but one already exists.`

I am guessing that this is due to
` .optOption("interOpNumThreads", sys.env.getOrElse("ONNX_INTEROP_NUM_THREADS", "1"))
.optOption("intraOpNumThreads", sys.env.getOrElse("ONNX_INTRAOP_NUM_THREADS", "1"))`
this being called multiple times.

Can you suggest what practice we should follow here?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.