facebookresearch / facebookresearch/SlowFast

Memory Error while loading Kinetic Dataset

Open
#551 10 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
7.4k
Forks
1.3k
PR merge metrics
No merged PRs in 30d

Description

Hi, I am trying to reproduce the results of the MViT model on the Kinetic dataset.
However, I encounter `double free or corruption (!prev)` and `malloc(): invalid size (unsorted)` errors while loading the Kinectic dataset

[06/01 12:11:45][INFO] kinetics.py: 88: Constructing Kinetics test...
[06/01 12:11:46][INFO] kinetics.py: 152: Constructing kinetics dataloader (size: 386710 skip_rows 0) from ../tmp_slowfast/cv2_data/test.csv
[06/01 12:11:46][INFO] test_net.py: 211: Testing model for 6043 iterations
double free or corruption (!prev)
malloc(): invalid size (unsorted)
double free or corruption (!prev)
Traceback (most recent call last):
File "/home/tduy/anaconda3/envs/slowfast/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 986, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/home/tduy/anaconda3/envs/slowfast/lib/python3.8/queue.py", line 179, in get
self.not_empty.wait(remaining)
File "/home/tduy/anaconda3/envs/slowfast/lib/python3.8/threading.py", line 306, in wait
gotit = waiter.acquire(True, timeout)
File "/home/tduy/anaconda3/envs/slowfast/lib/python3.8/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 4035577) is killed by signal: Aborted.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "tools/run_net.py", line 45, in
main()
File "tools/run_net.py", line 30, in main
launch_job(cfg=cfg, init_method=args.init_method, func=test)
File "/home/tduy/Workspace/git/newest_slowfast/slowfast/utils/misc.py", line 311, in launch_job
func(cfg=cfg)
File "/home/tduy/Workspace/git/newest_slowfast/tools/test_net.py", line 244, in test
test_meter = perform_test(test_loader, model, test_meter, cfg, writer)
File "/home/tduy/anaconda3/envs/slowfast/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/home/tduy/Workspace/git/newest_slowfast/tools/test_net.py", line 50, in perform_test
for cur_iter, (inputs, labels, video_idx, time, meta) in enumerate(
File "/home/tduy/anaconda3/envs/slowfast/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 517, in __next__
data = self._next_data()
File "/home/tduy/anaconda3/envs/slowfast/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1182, in _next_data
idx, data = self._get_data()
File "/home/tduy/anaconda3/envs/slowfast/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1138, in _get_data
success, data = self._try_get_data()
File "/home/tduy/anaconda3/envs/slowfast/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 999, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) from e
RuntimeError: DataLoader worker (pid(s) 4035577) exited unexpectedly

It seems that the error is related to the buffer allocation in the `decode` function inside the `kinetic.py` file.
Does anybody know how to fix this?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.