aboutcode-org / aboutcode-org/scancode-toolkit

License cache not always reused

オープン
#4,273 コメント 3 件 リアクション 1 件 担当者 0 名 GitHub で見る
bug
主要言語
Python
スター
2.6k
フォーク
791
平均マージ
1日 12時間
マージ済み PR(30日)
5

説明

### Description

The licensedcode cache is not always utilized when running multiple processes in parallel. This was noticed while running stress-tests on scancode. We observed that, when multiple tests were started in separate processes at the same time, each process would separately build its own cache instead of using the existing one. This had a considerable performance cost, eventually leading to a `LockTimeout`.

The root cause is in licensedcode/cache.py: After a process obtains a lock, it should check if another thread has already built the cache, but it does not.

### How To Reproduce

This was noticed when stress-testing a local test:
```
with NamedTemporaryFile() as test_file:
test_contents = bytes(MIT_LICENSE_TEXT.encode("utf-8"))
test_file.write(test_contents)
test_file.seek(0)
results = get_licenses(test_file.name) # slow
license_expression = results["detected_license_expression"]
self.assertEqual(license_expression, "mit")
```
We ran this on 100 processes in parallel.
```
Traceback (most recent call last):
File "scancode/api.py", line 200, in get_licenses
for detection in detections:
File "licensedcode/detection.py", line 1947, in detect_licenses
index = cache.get_index()
File "licensedcode/cache.py", line 459, in get_index
return get_cache(
File "licensedcode/cache.py", line 399, in get_cache
return populate_cache(
File "licensedcode/cache.py", line 419, in populate_cache
_LICENSE_CACHE = LicenseCache.load_or_build(
File "licensedcode/cache.py", line 136, in load_or_build
with lockfile.FileLock(lock_file).locked(timeout=timeout):
File "runtime/lib/python3.10/contextlib.py", line 135, in __enter__
return next(self.gen)
File "scancode/lockfile.py", line 29, in locked
raise LockTimeout(timeout)
scancode.lockfile.LockTimeout: 360
```

### System configuration

* OS: Linux
* What version of scancode-toolkit was used to generate the scan file? scancode-toolkit-mini 32.3.2
* What installation method was used to install/run scancode? pip

コントリビューションガイド

コントリビューションガイドを開く

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。