aboutcode-org / aboutcode-org/scancode-toolkit

Do not scan certain data-only data file

Đang mở
#602 5 bình luận 0 reaction 0 người được giao Xem trên GitHub
copyright scan core and api easy good first issue license scan
Ngôn ngữ chính
Python
Star
2.6k
Fork
791
Merge trung bình
1 ngày 12 giờ
Pull request đã merge (30 ngày)
5

Mô tả

I have mixed feelings on data files: I want to scan them as they may contain licenses or clues AND I do not want to scan them if they are highly likely not to contain any clues.

For the copyright detection that seeks possible dates and date ranges (such as Copyright (c) 2000-2013 XXXX), long list of numbers are a hog for now.
There are other cases where scanning data files is not great especially when these are big files.
Or for instance the firmwares are mostly hex blobs in the kernel. There I still want to scan them even if these may be huge hex blob-like: they have the most byzantine licenses of all and are worth reporting.

So we need a better way to skip scanning certain files that are pure data.
And in these cases just issue a warning that they were not scanned fully and likely only scan the first and last hundred lines.

There are a couple patterns like binary-only data files which are easy to ignore.
Or text files with lines made of digits, puncts and X (to catch hex and numbers lists).
I wished I could use a quick entropy computation, but these lines may have a high entropy too. But they have in some cases a fixed width format that is always the same and may be a clue to rely on: e.g. a formatted text where the format of each line itself is the same.

One example of data files (e.g. *.data) is in http://http.debian.net/debian/pool/main/a/ask/ask_1.0.1.orig.tar.gz
Other .hex files are found in the Linux kernel.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.