explosion / explosion/weasel

MemoryError on computing checksums for large files

Open
#79 2 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
93
Forks
15
PR merge metrics
No merged PRs in 30d

Description

When creating a command which depends on a large file (which cannot be fitted into memory), weasel still [tries to load the whole file](https://github.com/explosion/weasel/blob/main/weasel/util/hashing.py#L33) which results in a `MemoryError`.

The traceback for such a run:

```Traceback (most recent call last):

File "/home/gorosz/workspace/ascend/invoice-ai-models/.venv/bin/weasel", line 8, in sys.exit(app())

File "/home/gorosz/workspace/ascend/invoice-ai-models/.venv/lib/python3.10/site-packages/weasel/cli/run.py", line 42, in project_run_cli project_run(

File "/home/gorosz/workspace/ascend/invoice-ai-models/.venv/lib/python3.10/site-packages/weasel/cli/run.py", line 88, in project_run
project_run(
File "/home/gorosz/workspace/ascend/invoice-ai-models/.venv/lib/python3.10/site-packages/weasel/cli/run.py", line 113, in project_run
update_lockfile(current_dir, cmd)
File "/home/gorosz/workspace/ascend/invoice-ai-models/.venv/lib/python3.10/site-packages/weasel/cli/run.py", line 270, in update_lockfile
data[command["name"]] = get_lock_entry(project_dir, command)
File "/home/gorosz/workspace/ascend/invoice-ai-models/.venv/lib/python3.10/site-packages/weasel/cli/run.py", line 286, in get_lock_entry
deps = get_fileinfo(project_dir, command.get("deps", []))

File "/home/gorosz/workspace/ascend/invoice-ai-models/.venv/lib/python3.10/site-packages/weasel/cli/run.py", line 308, in get_fileinfo
md5 = get_checksum(file_path) if file_path.exists() else None

File "/home/gorosz/workspace/ascend/invoice-ai-models/.venv/lib/python3.10/site-packages/weasel/util/hashing.py", line 33, in get_checksum
return hashlib.md5(Path(path).read_bytes()).hexdigest()

File "/home/gorosz/Applications/miniconda3/lib/python3.10/pathlib.py", line 1127, in read_bytes
return f.read()

MemoryError
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.