deepjavalibrary / deepjavalibrary/djl
Improperly Defined or Garbage Collected Resources in Block.forward Method Call
- Dominant language
- Java
- Stars
- 4.9k
- Forks
- 759
- Avg merge
- 19h 26m
- Merged PRs (30d)
- 17
Description
## Description
After training a model for use as a variational autoencoder earlier in the code, I get an error related to resources as soon as I attempt to use that model following training. The error generated is "Native resource has been released already." My code makes no attempt to release resources and all of the code is executing within the scope of a single method so no resources should be going out of scope. The resources in question seem to be the weight and bias variables created in the Linear.java forwardInternal method. Details are provided in the "What have you tried to solve it?" section.
I pass a newly created ParameterStore object when calling the forward method, but if an alternative approach should be used instead, I would be happy to use that. I have not, however, found such an approach defined.
### Expected Behavior
I would expect resources to not be released unless there was an explicit action or a change of program scope that resulted in garbage collection or to be created properly so that they could be used. I would also expect variables to include some sort of flag to indicate their lifecycle with regard to whether they have been initialized previously. I would also expect some identification of the resource that has been released to be included in exception output.
### Error Message
```
Exception in thread "main" java.lang.IllegalStateException: Native resource has been released already.
at ai.djl.util.NativeResource.getHandle(NativeResource.java:64)
at ai.djl.pytorch.jni.JniUtils.linear(JniUtils.java:1411)
at ai.djl.pytorch.engine.PtNDArrayEx.linear(PtNDArrayEx.java:397)
at ai.djl.nn.core.Linear.linear(Linear.java:192)
at ai.djl.nn.core.Linear.forwardInternal(Linear.java:96)
at ai.djl.nn.AbstractBaseBlock.forward(AbstractBaseBlock.java:79)
at ai.djl.nn.Block.forward(Block.java:127)
at ai.djl.nn.SequentialBlock.forwardInternal(SequentialBlock.java:191)
at ai.djl.nn.AbstractBaseBlock.forward(AbstractBaseBlock.java:79)
at ai.djl.nn.Block.forward(Block.java:127)
```
## How to Reproduce?
Reproduction is difficult at this stage outside of my code as the code is part of a pipeline which makes creation of a stand alone block complicated.
### Steps to reproduce
I can test only in the context of my current code at present, but I am happy to test or investigate if suggestions are made on changes to make.
## What have you tried to solve it?
In following the traces, the resources that appear to be released or that were never allocated to begin with are the "weight" and "bias" variables. Both are showing a toString value of "This array is already closed". I have made no explicit allocation of such a parameter in my code, but the Linear.java forwardInternal method creates these variables (https://github.com/deepjavalibrary/djl/blob/c6d78bc9f4ae080eef70772a89682719ebb6db04/api/src/main/java/ai/djl/nn/core/Linear.java#L94). However they seem to be created in an invalid state via ParameterStore class as they immediately have toString values of "This array is already closed" upon creation within that method and later generate the exception.
## Environment Info
I don't have gradle on my system. I am using Windows 10 with DJL 0.31.0 and pytorch as the engine. It would be helpful if the template outlined which information you need here instead of expecting output of a build command.
Contributor guide
Research direction
Start with the stack trace and inspect Linear.java forwardInternal, ParameterStore, NativeResource.java, JniUtils.java, and PtNDArrayEx.java under the DJL 0.31.0 PyTorch setup. First reduce the pipeline to a standalone block if possible; done means reproducing the failure and determining why the weight or bias resource is already closed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100