[bug] There might be some memory leak

NOTE: ISSUES ARE NOT FOR CODE HELP - Ask for Help at https://stackoverflow.com

Your issue may already be reported!
Also, please search on the [issue tracker](../) before creating one.

* **I'm submitting a ...**
- [x] bug report
- [ ] feature request
- [ ] support request => Please do not submit support request here, see note at the top of this template.

# Issue Description
From [this PR](https://github.com/automl/Auto-PyTorch/runs/5123733321?check_suite_focus=true), we have failures in unittest for cross validation in tabular classification.
If my guess is correct, this is due to some kind of memory leak.
After we reach the phase of experiments, we need to fix this issue so that we will not have serious memory leak.