ResourceWarning: Unclosed file... messages when deserializing a model.
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.9k
- Forks
- 107
- PR merge metrics
- No merged PRs in 30d
Description
The load_obj in the pickle_utils.py file makes lots of use of this pattern:
`dill.load(open(os.path.join(dir_name, file), 'rb'))`
And since dill does not close the file handler passed to the `load` method, it causes a Resource Warning for each of the parts of the model being loaded. In the following case, it caused 2856 of those warnings flooding the logs.
```
c:\windows\serviceprofiles\networkservice\.conda\envs\aml_test_38\lib\site-packages\greykite\framework\templates\pickle_utils.py:313: ResourceWarning:
2022-08-20T16:56:02.4762240Z
2022-08-20T16:56:02.4762597Z unclosed file <_io.BufferedReader name='C:\\Windows\\SERVIC~2\\NETWOR~1\\AppData\\Local\\Temp\\tmp55hy74nb\\tmp7q08pus4.type'>
2022-08-20T16:56:02.4762842Z
2022-08-20T16:56:02.4763280Z c:\windows\serviceprofiles\networkservice\.conda\envs\aml_test_38\lib\site-packages\greykite\framework\templates\pickle_utils.py:313: ResourceWarning:
2022-08-20T16:56:02.4764328Z
2022-08-20T16:56:02.4764749Z unclosed file <_io.BufferedReader name='C:\\Windows\\SERVIC~2\\NETWOR~1\\AppData\\Local\\Temp\\tmp55hy74nb\\tmp7q08pus4\\df_dropna__value__.type'>
2022-08-20T16:56:02.4765023Z
2022-08-20T16:56:02.4765471Z c:\windows\serviceprofiles\networkservice\.conda\envs\aml_test_38\lib\site-packages\greykite\framework\templates\pickle_utils.py:313: ResourceWarning:
2022-08-20T16:56:02.4765803Z
2022-08-20T16:56:02.4766176Z unclosed file <_io.BufferedReader name='C:\\Windows\\SERVIC~2\\NETWOR~1\\AppData\\Local\\Temp\\tmp55hy74nb\\tmp7q08pus4\\df__value__.type'>
2022-08-20T16:56:02.4766440Z
2022-08-20T16:56:02.4766859Z c:\windows\serviceprofiles\networkservice\.conda\envs\aml_test_38\lib\site-packages\greykite\framework\templates\pickle_utils.py:313: ResourceWarning:
2022-08-20T16:56:02.4767206Z
2022-08-20T16:56:02.4767571Z unclosed file <_io.BufferedReader name='C:\\Windows\\SERVIC~2\\NETWOR~1\\AppData\\Local\\Temp\\tmp55hy74nb\\tmp7q08pus4\\features_df__value__.type'>
2022-08-20T16:56:02.4767866Z
2022-08-20T16:56:02.4768299Z c:\windows\serviceprofiles\networkservice\.conda\envs\aml_test_38\lib\site-packages\greykite\framework\templates\pickle_utils.py:313: ResourceWarning:
2022-08-20T16:56:02.4768626Z
2022-08-20T16:56:02.4769008Z unclosed file <_io.BufferedReader name='C:\\Windows\\SERVIC~2\\NETWOR~1\\AppData\\Local\\Temp\\tmp55hy74nb\\tmp7q08pus4\\fitted_df__value__.type'>
2022-08-20T16:56:02.4769279Z
2022-08-20T16:56:02.4769714Z c:\windows\serviceprofiles\networkservice\.conda\envs\aml_test_38\lib\site-packages\greykite\framework\templates\pickle_utils.py:313: ResourceWarning:
2022-08-20T16:56:02.4770043Z
2022-08-20T16:56:02.4770431Z unclosed file <_io.BufferedReader name='C:\\Windows\\SERVIC~2\\NETWOR~1\\AppData\\Local\\Temp\\tmp55hy74nb\\tmp7q08pus4\\fs_components_df__value__.type'>
2022-08-20T16:56:02.4770714Z
2022-08-20T16:56:02.4771148Z c:\windows\serviceprofiles\networkservice\.conda\envs\aml_test_38\lib\site-packages\greykite\framework\templates\pickle_utils.py:313: ResourceWarning:
2022-08-20T16:56:02.4771476Z
2022-08-20T16:56:02.4771863Z unclosed file <_io.BufferedReader name='C:\\Windows\\SERVIC~2\\NETWOR~1\\AppData\\Local\\Temp\\tmp55hy74nb\\tmp7q08pus4\\ml_model_summary__value__.type'>
2022-08-20T16:56:02.4772143Z
2022-08-20T16:56:02.4772588Z c:\windows\serviceprofiles\networkservice\.conda\envs\aml_test_38\lib\site-packages\greykite\framework\templates\pickle_utils.py:313: ResourceWarning:
2022-08-20T16:56:02.4772917Z
2022-08-20T16:56:02.4773290Z unclosed file <_io.BufferedReader name='C:\\Windows\\SERVIC~2\\NETWOR~1\\AppData\\Local\\Temp\\tmp55hy74nb\\tmp7q08pus4\\uncertainty_model__value__.type'>
2022-08-20T16:56:02.4773589Z
2022-08-20T16:56:02.4774004Z c:\windows\serviceprofiles\networkservice\.conda\envs\aml_test_38\lib\site-packages\greykite\framework\templates\pickle_utils.py:313: ResourceWarning:
2022-08-20T16:56:02.4774354Z
2022-08-20T16:56:02.4774721Z unclosed file <_io.BufferedReader name='C:\\Windows\\SERVIC~2\\NETWOR~1\\AppData\\Local\\Temp\\tmp55hy74nb\\tmp7q08pus4\\x_design_info__value__.type'>
2022-08-20T16:56:02.4775010Z
2022-08-20T16:56:02.4775425Z c:\windows\serviceprofiles\networkservice\.conda\envs\aml_test_38\lib\site-packages\greykite\framework\templates\pickle_utils.py:313: ResourceWarning:
2022-08-20T16:56:02.4776102Z
2022-08-20T16:56:02.4776491Z unclosed file <_io.BufferedReader name='C:\\Windows\\SERVIC~2\\NETWOR~1\\AppData\\Local\\Temp\\tmp55hy74nb\\tmp7q08pus4\\x_mat__value__.type'>
2022-08-20T16:56:02.4776755Z
2022-08-20T16:56:02.4777188Z c:\windows\serviceprofiles\networkservice\.conda\envs\aml_test_38\lib\site-packages\greykite\framework\templates\pickle_utils.py:376: ResourceWarning:
2022-08-20T16:56:02.4777512Z
2022-08-20T16:56:02.4777900Z unclosed file <_io.BufferedReader name='C:\\Windows\\SERVIC~2\\NETWOR~1\\AppData\\Local\\Temp\\tmp55hy74nb\\tmp7q08pus4\\adjust_anomalous_info__key__.pkl'>
2022-08-20T16:56:02.4778437Z
2022-08-20T16:56:02.4778891Z c:\windows\serviceprofiles\networkservice\.conda\envs\aml_test_38\lib\site-packages\greykite\framework\templates\pickle_utils.py:395: ResourceWarning:
2022-08-20T16:56:02.4779216Z
2022-08-20T16:56:02.4779615Z unclosed file <_io.BufferedReader name='C:\\Windows\\SERVIC~2\\NETWOR~1\\AppData\\Local\\Temp\\tmp55hy74nb\\tmp7q08pus4\\adjust_anomalous_info__value__.pkl'>
...
```
We were using graykite 0.2.0, but I see the issue is still present in version 0.4.0.
If instead we had a simple `read_pickle` or `dill_load` function that would handle closing the file:
```
def read_pickle(path, mode):
"""Loads the pickled files and closes the file handle.
Parameters
----------
path : `str`
The path to the pickled file.
mode : `str`
The mode of the open function.
"""
with open(path, mode) as file:
data = dill.load(file)
return data
```
It would be just a question of changing the calls to `dill.load(open(...))` to `read_pickle(...)`. Are there any reason not to use this pattern?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in pickle_utils.py, especially load_obj and the dill.load(open(...)) calls described in the issue. Trace the model deserialization paths that open files, then verify that loading a model closes each file handle and no longer emits the reported ResourceWarning messages.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100