TypeError: cannot serialize '_io.BufferedWriter' object when copy action dict in get_memory_handler

## Type of Issue

Select the type of issue:
- [ ] Bug report (to report a bug)
- [ ] Feature request (to request an additional feature)
- [ ] Tracker (I am just using this as a tracker)
- [ ] Refactor request
- [ ] Documentation Ask

## Description

```
INFO:root:Incoming chats: [' what is the height of the cube']
2021-05-12 12:23:47,489 [loco_mc_agent.py:256 - controller_step() INFO]: Incoming chats: [' what is the height of the cube']
DEBUG:root:Incoming chat: ['1615933d30808eea34454d9a89d07f8a0d3f3470f181b9b7072cf06f3c3d60fc:0815629c72f245bbbb31a6694fb70b0f' -> what is the height of the cube]
2021-05-12 12:23:47,490 [loco_mc_agent.py:266 - controller_step() DEBUG]: Incoming chat: ['1615933d30808eea34454d9a89d07f8a0d3f3470f181b9b7072cf06f3c3d60fc:0815629c72f245bbbb31a6694fb70b0f' -> what is the height of the cube]
DEBUG:root:all_nearby_objects found 1 objects near [ 0 63 0]
2021-05-12 12:23:47,791 [heuristic_perception.py:58 - all_nearby_objects() DEBUG]: all_nearby_objects found 1 objects near [ 0 63 0]
DEBUG:root:Dialogue stack pre-run_model: []
2021-05-12 12:23:48,099 [dialogue_manager.py:81 - step() DEBUG]: Dialogue stack pre-run_model: []
INFO:root:Querying the semantic parsing model
2021-05-12 12:23:48,100 [dialogue_model.py:65 - query_for_logical_form() INFO]: Querying the semantic parsing model
DEBUG:root:chat before lemmatization "what is the height of the cube"
2021-05-12 12:23:49,008 [droidlet_nsp_model_wrapper.py:195 - postprocess_logical_form() DEBUG]: chat before lemmatization "what is the height of the cube"
DEBUG:root:chat after lemmatization "what be the height of the cube"
2021-05-12 12:23:49,045 [droidlet_nsp_model_wrapper.py:198 - postprocess_logical_form() DEBUG]: chat after lemmatization "what be the height of the cube"
DEBUG:root:ttad pre-coref "what be the height of the cube" -> {'dialogue_type': 'GET_MEMORY', 'filters': {'triples': [{'pred_text': 'has_name', 'obj_text': 'cube'}], 'output': {'attribute': 'SIZE'}, 'contains_coreference': 'yes', 'memory_type': 'REFERENCE_OBJECT'}}
2021-05-12 12:23:49,046 [droidlet_nsp_model_wrapper.py:219 - postprocess_logical_form() DEBUG]: ttad pre-coref "what be the height of the cube" -> {'dialogue_type': 'GET_MEMORY', 'filters': {'triples': [{'pred_text': 'has_name', 'obj_text': 'cube'}], 'output': {'attribute': 'SIZE'}, 'contains_coreference': 'yes', 'memory_type': 'REFERENCE_OBJECT'}}
DEBUG:root:logical form post-coref "399acc6d883aa123edbb72bf88f2a2d24dd28d88cc322fa6f59e0ae08c4d352b:78f4eb89724a4b1ea2c5534c3385bf87" -> {'dialogue_type': 'GET_MEMORY', 'filters': {'triples': [{'pred_text': 'has_name', 'obj_text': 'cube'}], 'output': {'attribute': 'SIZE'}, 'contains_coreference': , 'memory_type': 'REFERENCE_OBJECT'}}
2021-05-12 12:23:49,050 [droidlet_nsp_model_wrapper.py:225 - postprocess_logical_form() DEBUG]: logical form post-coref "399acc6d883aa123edbb72bf88f2a2d24dd28d88cc322fa6f59e0ae08c4d352b:78f4eb89724a4b1ea2c5534c3385bf87" -> {'dialogue_type': 'GET_MEMORY', 'filters': {'triples': [{'pred_text': 'has_name', 'obj_text': 'cube'}], 'output': {'attribute': 'SIZE'}, 'contains_coreference': , 'memory_type': 'REFERENCE_OBJECT'}}
ERROR:root:Default handler caught exception, db_log_idx=46807
Traceback (most recent call last):
File "/private/home/yuxuans/Workspace/tmp/droidlet/base_agent/core.py", line 19, in start
self.step()
File "/private/home/yuxuans/Workspace/tmp/droidlet/base_agent/loco_mc_agent.py", line 213, in step
super().step()
File "/private/home/yuxuans/Workspace/tmp/droidlet/base_agent/core.py", line 27, in step
self.controller_step()
File "/private/home/yuxuans/Workspace/tmp/droidlet/base_agent/loco_mc_agent.py", line 279, in controller_step
self.dialogue_manager.step(incoming_chats[0])
File "/private/home/yuxuans/Workspace/tmp/droidlet/base_agent/dialogue_manager.py", line 87, in step
obj = self.semantic_parsing_model_wrapper.get_dialogue_object()
File "/private/home/yuxuans/Workspace/tmp/droidlet/base_agent/droidlet_nsp_model_wrapper.py", line 170, in get_dialogue_object
speaker=speaker, logical_form=updated_logical_form, chat=chat
File "/private/home/yuxuans/Workspace/tmp/droidlet/base_agent/droidlet_nsp_model_wrapper.py", line 306, in handle_logical_form
speaker, logical_form, **self.dialogue_object_parameters
File "/private/home/yuxuans/Workspace/tmp/droidlet/craftassist/agent/dialogue_objects/get_memory_handler.py", line 36, in __init__
super().__init__(speaker_name, action_dict, **kwargs)
File "/private/home/yuxuans/Workspace/tmp/droidlet/base_agent/dialogue_objects/get_memory_handler.py", line 36, in __init__
self.action_dict_orig = deepcopy(self.action_dict)
File "/private/home/yuxuans/.conda/envs/minecraft_env/lib/python3.7/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/private/home/yuxuans/.conda/envs/minecraft_env/lib/python3.7/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/private/home/yuxuans/.conda/envs/minecraft_env/lib/python3.7/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/private/home/yuxuans/.conda/envs/minecraft_env/lib/python3.7/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/private/home/yuxuans/.conda/envs/minecraft_env/lib/python3.7/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/private/home/yuxuans/.conda/envs/minecraft_env/lib/python3.7/copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "/private/home/yuxuans/.conda/envs/minecraft_env/lib/python3.7/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/private/home/yuxuans/.conda/envs/minecraft_env/lib/python3.7/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/private/home/yuxuans/.conda/envs/minecraft_env/lib/python3.7/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/private/home/yuxuans/.conda/envs/minecraft_env/lib/python3.7/copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "/private/home/yuxuans/.conda/envs/minecraft_env/lib/python3.7/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/private/home/yuxuans/.conda/envs/minecraft_env/lib/python3.7/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/private/home/yuxuans/.conda/envs/minecraft_env/lib/python3.7/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/private/home/yuxuans/.conda/envs/minecraft_env/lib/python3.7/copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "/private/home/yuxuans/.conda/envs/minecraft_env/lib/python3.7/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/private/home/yuxuans/.conda/envs/minecraft_env/lib/python3.7/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/private/home/yuxuans/.conda/envs/minecraft_env/lib/python3.7/copy.py", line 169, in deepcopy
rv = reductor(4)
TypeError: cannot serialize '_io.BufferedWriter' object
2021-05-12 12:23:49,052 [loco_mc_agent.py:189 - handle_exception() ERROR]: Default handler caught exception, db_log_idx=46807
Traceback (most recent call last):
File "/private/home/yuxuans/Workspace/tmp/droidlet/base_agent/core.py", line 19, in start
self.step()
File "/private/home/yuxuans/Workspace/tmp/droidlet/base_agent/loco_mc_agent.py", line 213, in step
super().step()
File "/private/home/yuxuans/Workspace/tmp/droidlet/base_agent/core.py", line 27, in step
self.controller_step()
File "/private/home/yuxuans/Workspace/tmp/droidlet/base_agent/loco_mc_agent.py", line 279, in controller_step
self.dialogue_manager.step(incoming_chats[0])
File "/private/home/yuxuans/Workspace/tmp/droidlet/base_agent/dialogue_manager.py", line 87, in step
obj = self.semantic_parsing_model_wrapper.get_dialogue_object()
File "/private/home/yuxuans/Workspace/tmp/droidlet/base_agent/droidlet_nsp_model_wrapper.py", line 170, in get_dialogue_object
speaker=speaker, logical_form=updated_logical_form, chat=chat
File "/private/home/yuxuans/Workspace/tmp/droidlet/base_agent/droidlet_nsp_model_wrapper.py", line 306, in handle_logical_form
speaker, logical_form, **self.dialogue_object_parameters
File "/private/home/yuxuans/Workspace/tmp/droidlet/craftassist/agent/dialogue_objects/get_memory_handler.py", line 36, in __init__
super().__init__(speaker_name, action_dict, **kwargs)
File "/private/home/yuxuans/Workspace/tmp/droidlet/base_agent/dialogue_objects/get_memory_handler.py", line 36, in __init__
self.action_dict_orig = deepcopy(self.action_dict)
File "/private/home/yuxuans/.conda/envs/minecraft_env/lib/python3.7/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/private/home/yuxuans/.conda/envs/minecraft_env/lib/python3.7/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/private/home/yuxuans/.conda/envs/minecraft_env/lib/python3.7/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/private/home/yuxuans/.conda/envs/minecraft_env/lib/python3.7/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/private/home/yuxuans/.conda/envs/minecraft_env/lib/python3.7/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/private/home/yuxuans/.conda/envs/minecraft_env/lib/python3.7/copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "/private/home/yuxuans/.conda/envs/minecraft_env/lib/python3.7/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/private/home/yuxuans/.conda/envs/minecraft_env/lib/python3.7/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/private/home/yuxuans/.conda/envs/minecraft_env/lib/python3.7/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/private/home/yuxuans/.conda/envs/minecraft_env/lib/python3.7/copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "/private/home/yuxuans/.conda/envs/minecraft_env/lib/python3.7/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/private/home/yuxuans/.conda/envs/minecraft_env/lib/python3.7/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/private/home/yuxuans/.conda/envs/minecraft_env/lib/python3.7/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/private/home/yuxuans/.conda/envs/minecraft_env/lib/python3.7/copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "/private/home/yuxuans/.conda/envs/minecraft_env/lib/python3.7/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/private/home/yuxuans/.conda/envs/minecraft_env/lib/python3.7/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/private/home/yuxuans/.conda/envs/minecraft_env/lib/python3.7/copy.py", line 169, in deepcopy
rv = reductor(4)
TypeError: cannot serialize '_io.BufferedWriter' object
Traceback (most recent call last):
File "craftassist_agent.py", line 336, in
sa.start()
File "/private/home/yuxuans/Workspace/tmp/droidlet/base_agent/core.py", line 21, in start
self.handle_exception(e)
File "/private/home/yuxuans/Workspace/tmp/droidlet/base_agent/loco_mc_agent.py", line 208, in handle_exception
raise e
File "/private/home/yuxuans/Workspace/tmp/droidlet/base_agent/core.py", line 19, in start
self.step()
File "/private/home/yuxuans/Workspace/tmp/droidlet/base_agent/loco_mc_agent.py", line 213, in step
super().step()
File "/private/home/yuxuans/Workspace/tmp/droidlet/base_agent/core.py", line 27, in step
self.controller_step()
File "/private/home/yuxuans/Workspace/tmp/droidlet/base_agent/loco_mc_agent.py", line 279, in controller_step
self.dialogue_manager.step(incoming_chats[0])
File "/private/home/yuxuans/Workspace/tmp/droidlet/base_agent/dialogue_manager.py", line 87, in step
obj = self.semantic_parsing_model_wrapper.get_dialogue_object()
File "/private/home/yuxuans/Workspace/tmp/droidlet/base_agent/droidlet_nsp_model_wrapper.py", line 170, in get_dialogue_object
speaker=speaker, logical_form=updated_logical_form, chat=chat
File "/private/home/yuxuans/Workspace/tmp/droidlet/base_agent/droidlet_nsp_model_wrapper.py", line 306, in handle_logical_form
speaker, logical_form, **self.dialogue_object_parameters
File "/private/home/yuxuans/Workspace/tmp/droidlet/craftassist/agent/dialogue_objects/get_memory_handler.py", line 36, in __init__
super().__init__(speaker_name, action_dict, **kwargs)
File "/private/home/yuxuans/Workspace/tmp/droidlet/base_agent/dialogue_objects/get_memory_handler.py", line 36, in __init__
self.action_dict_orig = deepcopy(self.action_dict)
File "/private/home/yuxuans/.conda/envs/minecraft_env/lib/python3.7/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/private/home/yuxuans/.conda/envs/minecraft_env/lib/python3.7/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/private/home/yuxuans/.conda/envs/minecraft_env/lib/python3.7/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/private/home/yuxuans/.conda/envs/minecraft_env/lib/python3.7/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/private/home/yuxuans/.conda/envs/minecraft_env/lib/python3.7/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/private/home/yuxuans/.conda/envs/minecraft_env/lib/python3.7/copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "/private/home/yuxuans/.conda/envs/minecraft_env/lib/python3.7/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/private/home/yuxuans/.conda/envs/minecraft_env/lib/python3.7/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/private/home/yuxuans/.conda/envs/minecraft_env/lib/python3.7/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/private/home/yuxuans/.conda/envs/minecraft_env/lib/python3.7/copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "/private/home/yuxuans/.conda/envs/minecraft_env/lib/python3.7/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/private/home/yuxuans/.conda/envs/minecraft_env/lib/python3.7/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/private/home/yuxuans/.conda/envs/minecraft_env/lib/python3.7/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/private/home/yuxuans/.conda/envs/minecraft_env/lib/python3.7/copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "/private/home/yuxuans/.conda/envs/minecraft_env/lib/python3.7/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/private/home/yuxuans/.conda/envs/minecraft_env/lib/python3.7/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/private/home/yuxuans/.conda/envs/minecraft_env/lib/python3.7/copy.py", line 169, in deepcopy
rv = reductor(4)
TypeError: cannot serialize '_io.BufferedWriter' object
```

## Current Behavior

Description of current behavior or functionality.

## Expected Behavior

Description of the expected behavior or functionality. In case of feature request, please add input and expected output.

## Steps to reproduce

Please add steps to reproduce the bug here along with the stack trace.

## Links to any relevant pastes or documents

Please post links to any relevant documents that might contain any extra information.

## Checklist

Use this checklist if you are using this issue as a tracker:

- [ ] Task 1
- [ ] Task 2
- [ ] Task 3 ...