multi-gigabyte memory-spikes when processing large files

### Description

Normally `scancode` is quite predictable in its memory use. It sits around 1GB per process used (`--processes`), the absolute majority of the memory being occupied by the rule index. However, when large files (1MB+) are encountered memory consumption becomes much less predictable and can spike to more than twice that amount _per process_ (in [rare cases](https://github.com/nexB/scancode-toolkit/issues/3711#issuecomment-2049500920) it can explode to 12GB for one individual file!).

I cannot judge whether this is due to some algorithmic inefficiency or would require a new approach such as chunked processing of files. Either way it would be nice to have memory consumption more predictable, as it has been found to result in out-of-memory kills.

Normal memory use can be illustrated by running `scancode` against [go-git](https://github.com/go-git/go-git/tree/v5.0.0), which only contains small files (<10Kb). As can be seen in the graph below memory use is _very stable_:

```shell
cd /tmp
git clone https://github.com/go-git/go-git && cd go-git
git checkout -b v5_0_0 tags/v5.0.0
scancode --json-pp scan.json -n 0 --timeout 600 --license --license-text --license-references /tmp/go-git/
```


However, when large files (1MB+) are encountered memory consumption becomes much less predictable and can spike to more than twice that amount (**update: in [rare cases](https://github.com/nexB/scancode-toolkit/issues/3711#issuecomment-2049500920) I've seen it spike at almost `12GB` on a file!**) . [This directory](https://github.com/torvalds/linux/tree/v6.7/drivers/gpu/drm/amd/include/asic_reg/dcn) from the Linux kernel is a good example of where `scancode` struggles to be memory-efficient. Notably [dcn_3_2_0_sh_mask.h](https://github.com/torvalds/linux/blob/v6.7/drivers/gpu/drm/amd/include/asic_reg/dcn/dcn_3_2_0_sh_mask.h), a 22MB, file, will cause `scancode` to spike at 2.2GB.

```shell
cd /tmp
git clone https://github.com/torvalds/linux && cd linux
git checkout -b v6_7 tags/v6.7
scancode --json-pp scan.json -n 0 --timeout 600 --license --license-text --license-references /tmp/linux/drivers/gpu/drm/amd/include/asic_reg
```


So that was with _one single process_. Now, consider running with `--processes=8` or similar and encountering many files like these. You can imagine the wild _total memory spikes_ you might end up with.

~~One shouldn't rule out there also being memory leaks, but at least it seems like most of the memory spikes get reclaimed (although memory never seems to drop all the way back to 1.0GB).~~(**update: in rare cases like [these](https://github.com/nexB/scancode-toolkit/issues/3711#issuecomment-2049500920), which spiked at `12GB` the memory did not appear to be reclaimed, suggesting a memory leak**)

### How To Reproduce

To reproduce one case where memory balloons, try the following:

```shell
python -m venv .venv
. .venv/bin/activate
pip install scancode-toolkit==32.0.8

wget https://raw.githubusercontent.com/torvalds/linux/v6.7/drivers/gpu/drm/amd/include/asic_reg/dcn/dcn_3_2_0_sh_mask.h
scancode --json-pp scan.json -n 0 --timeout 600 --license --license-text --license-references dcn_3_2_0_sh_mask.h
```

You should observe `scancode` consuming close to 2.2GB of memory for the single process that is working.

### System configuration

> - What OS are you running on? (Windows/MacOS/Linux)

Linux.

> - What version of scancode-toolkit was used to generate the scan file?

`32.0.8`

> - What installation method was used to install/run scancode? (pip/source download/other)

`pip`