mapbox / mapbox/robosat

RuntimeError: unexpected EOF, expected 2051578 more bytes. The file might be corrupted.

Open
#223 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
2.1k
Forks
385
PR merge metrics
No merged PRs in 30d

Description

Hello! @daniel-j-h Thank you for your project.

I am having an error during prediction:

`sudo docker run --runtime=nvidia -it --rm -v $PWD:/data --ipc=host mapbox/robosat:latest-gpu predict --tile_size 512 --model /data/robosat/config/model-unet.toml --dataset /data/robosat/config/dataset-building.toml --checkpoint /data/robosat/checkpoint/checkpoint-00001-of-00010.pth /data/robosat/datasets /data/`
```
Traceback (most recent call last):
File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/usr/src/app/robosat/tools/__main__.py", line 58, in
args.func(args)
File "/usr/src/app/robosat/tools/predict.py", line 60, in main
chkpt = torch.load(args.checkpoint, map_location=map_location)
File "/opt/venv/lib/python3.6/site-packages/torch/serialization.py", line 387, in load
return _load(f, map_location, pickle_module, **pickle_load_args)
File "/opt/venv/lib/python3.6/site-packages/torch/serialization.py", line 581, in _load
deserialized_objects[key]._set_from_file(f, offset, f_should_read_directly)
RuntimeError: unexpected EOF, expected 2051578 more bytes. The file might be corrupted.
terminate called after throwing an instance of 'c10::Error'
what(): owning_ptr == NullType::singleton() || owning_ptr->refcount_.load() > 0 ASSERT FAILED at /pytorch/c10/util/intrusive_ptr.h:350, please report a bug to PyTorch. intrusive_ptr: Can only intrusive_ptr::reclaim() owning pointers that were created using intrusive_ptr::release(). (reclaim at /pytorch/c10/util/intrusive_ptr.h:350)
frame #0: std::function::operator()() const + 0x11 (0x7fb856a7d441 in /opt/venv/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #1: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x2a (0x7fb856a7cd7a in /opt/venv/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #2: THStorage_free + 0xca (0x7fb857a1729a in /opt/venv/lib/python3.6/site-packages/torch/lib/libcaffe2.so)
frame #3: + 0x53a157 (0x7fb89f421157 in /opt/venv/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
frame #4: python3() [0x54f1c6]
frame #5: python3() [0x573420]
frame #6: python3() [0x4b1a48]
frame #7: python3() [0x5891e8]
frame #8: python3() [0x5add18]
frame #9: python3() [0x5add2e]
frame #10: python3() [0x5add2e]
frame #11: python3() [0x5add2e]
frame #12: python3() [0x5add2e]
frame #13: python3() [0x5add2e]
frame #14: python3() [0x56b906]

frame #20: __libc_start_main + 0xe7 (0x7fb8c8cbbb97 in /lib/x86_64-linux-gnu/libc.so.6)

Aborted (core dumped)
```
I am running on my laptop, with GTX 1660TI. Do you think this is the reason for that? Furthermore, getting same error while trying with only CPU.
Can you please comment on that?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with robosat/tools/predict.py at line 60, where torch.load reads the supplied checkpoint, and run the reported Docker prediction command. Check the checkpoint file and compare the failure on GPU and CPU; done means identifying whether the checkpoint is incomplete or the loading path is at fault and documenting a reproducible resolution.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.