Preparation for packaging scancode-toolkit for Debian

If I can find some funding for it, I would like to package scancode-toolkit for Debian, so that Debian contributors can use it to reduce their workload by automating copyright documentation creation/updates while creating new packages and updating existing packages. There are already some [tools](https://wiki.debian.org/CopyrightReviewTools) for this in Debian but I am hoping that scancode is better than them.

In preparation, I went through all the dependencies that get installed by [pip](https://scancode-toolkit.readthedocs.io/en/latest/getting-started/install.html#pip-install) when installing scancode and found that this list of PyPI packages are missing from Debian. I've annotated the list with some info and suggestions about these packages. A lot of them are forks or wrappers or otherwise unnecessary in Debian or for the use-case above. This issue aims to document all of those issues, track them and get them resolved so that packaging scancode-toolkit for Debian becomes more straightforward.

* [commoncode](https://pypi.org/project/commoncode/)
* created by nexB

* [container-inspector](https://pypi.org/project/container-inspector/)
* created by nexB
* not needed for Debian use-case
* make it optional?
* Philippe's comment: This is used here to support distro detection in rootfs, containers and VMs. (And also used in scancode.io)

* [debian-inspector](https://pypi.org/project/debian-inspector/)
* created by nexB
* mashup of several other packages
* created only because of licensing, however python-debian is GPLv2+, which via GPLv3 is [compatible](https://www.apache.org/licenses/GPL-compatibility.html) with Apache 2.0
* replace it with python-debian or add that as an option?
* Philippe's comment: Licensing was not the only reason this was created. The actual semantics of the way things are processed internally are different from the way python-debian handles things, in particular wrt. a more lenient approach to parsing, so making this compatible with python-debian would require major surgery which is IMHO way beyond a goal of porting ScanCode to Debian?

* [dockerfile-parse](https://pypi.org/project/dockerfile-parse/)
* not needed for Debian use-case
* make it optional?
* Philippe's comment: This is actually not used at all in ScanCode Toolkit proper but only container-inspector

* [dparse2](https://pypi.org/project/dparse2/)
* nexB fork of [dparse](https://pypi.org/project/dparse/)
* due to lack of maintainence
* upstream now maintained
* suggest merging upstream and switching
* Philippe's comment: I used to be a committer upstream and this is still unfixed security issues. dparse2 has diverged majorly from https://pypi.org/project/dparse/ and the few bits that subsists will eventually be removed or merged in scancode proper at some point of time. I have no inclination nor the time to merge and switch given the lack of historical responsiveness, security issues and overall lack of tests.

* [extractcode](https://pypi.org/project/extractcode/)
* created by nexB

* [extractcode-7z](https://pypi.org/project/extractcode-7z/)
* tiny wrapper around 7z
* suggest making it optional and make scancode look in `$PATH`
* Philippe's comment: there is a plugin architecture that can be set to use a distro-provided 7z

* [extractcode-libarchive](https://pypi.org/project/extractcode-libarchive/)
* tiny wrapper around libarchive C library
* scancode-plugins contains embedded code copies and prebuilt files so not packageable
* suggest allowing [libarchive-c](https://pypi.org/project/libarchive-c/) as already in Debian
* other PyPI libarchive wrappers could be used instead
* Philippe's comment: there is a plugin architecture that can be set to use a distro-provided libarchive. I have been contributing to libarchive-c in the past but the approach we use in https://github.com/nexB/extractcode/blob/main/src/extractcode/libarchive2.py to wrap is different and I doubt we can plug libarchive-c in without a major surgery.

https://github.com/nexB/extractcode/blob/main/src/extractcode/libarchive2.py

* [gemfileparser2](https://pypi.org/project/gemfileparser2/)
* nexB fork of [gemfileparser](https://pypi.org/project/gemfileparser/)
* maintainer active elsewhere
* suggest reaching out to discuss merging changes
* Philippe's comment: I am the primary maintainer of this package upstream and until recently I was missing the rights to make releases. I do have these rights now so the goal is to remerge indeed. Beyond this, we have a better Ruby gempspec parser in the works that may may this library obsolete for our use case.

* [jsonstreams](https://pypi.org/project/jsonstreams/)
* no recent commits
* maintainer active elsewhere
* suggest reaching out to offer funding for issues/PRs/etc
* Philippe's comment: this is a small and stable library. We have contributed bug fixes.
I am not sure there is anything to do there.

* [packageurl-python](https://pypi.org/project/packageurl-python/)
* funded
* many issues
* one PR
* otherwise fine

* [packvers](https://pypi.org/project/packvers/)
* nexB fork of [packaging](https://pypi.org/project/packaging/)
* looks like it was necessary though
* suggest frequent rebasing/merging
* Philippe's comment: I discussed with Pypa maintainers and we will likely switch to https://github.com/di/packaging_legacy instead of this (the "packaging" changes that led to create these libraries has been a major source of churn BTW ;) )

* [parameter-expansion-patched](https://pypi.org/project/parameter-expansion-patched/)
* Homepage link needs fixing to [this](https://github.com/nexB/parameter_expansion_patched)
* nexB fork of [parameter-expansion](https://pypi.org/project/parameter-expansion/)
* upstream now more maintained than fork
* suggest getting the one commit merged upstream and switch to that
* Philippe's comment: I am a committer upstream and the patched version was because of limited release rights. I know I have full rights and the intent is to merge back.

* [pip-requirements-parser](https://pypi.org/project/pip-requirements-parser/)
* nexB fork of pip
* suggest talking to pip about providing supported API that does what you want and is stable, switching to that
* possibly as a library external to pip
* suggest frequent rebasing/merging
* Philippe's comment: this library is now used in several places including cyclonedx, pip-audit and by pip maintainers and is here to stay as the only mostly correct pip requirements parser ;). pip maintainers do not/cannot expose an API for requirements otherwise in the current state of the tool.

* [pkginfo2](https://pypi.org/project/pkginfo2/)
* nexB fork of [pkginfo](https://pypi.org/project/pkginfo/)
* upstream more maintained than fork
* suggest talking to upstream about security requirements etc
* Philippe's comment: I submitted a few improvements and requirements upstream alright, but upstream has insecure defaults that are not being updated and we cannot therefore use upstream short of patching which is what we do.

* [plugincode](https://pypi.org/project/plugincode/)
* created by nexB
* wonder if there is a more widely used library to replace this
* Philippe's comment: this describes a the way you can plugin and extend ScanCode scanning stages. I doubt any other library would be able to do this. ;)

* [pygmars](https://pypi.org/project/pygmars/)
* nexB fork of [NLTK](https://pypi.org/project/nltk/)
* suggest discussing fork with upstream, maybe they have some ideas
* suggest frequent rebasing/merging
* Philippe's comment: this library resembles not much of anything from NLTK and will stay as it is as the new code helped making major performance improvement and is too far from NLTK original code to be of any value to upstream anymore. Beyond the code having gone through major refactoring, another issue with NLTK is that it had too many features for the limited using in ScanCode.

* [pymaven-patch](https://pypi.org/project/pymaven-patch/)
* Philippe's comment: upstream is not too maintained and ScanCode is the main user for this. We have several extensions in ScanCode proper that will need to move to our fork and I am not sure we can ever merge this back upstream . If this is possible we will.

* [scancode-toolkit](https://pypi.org/project/scancode-toolkit/)
* created by nexB
* vendors [attrs](https://pypi.org/project/attrs/)
* suggest devendoring and encoding broken versions in dependencies
* suggest discussing the issue upstream
* Philippe's comment: There is nothing broken in attrs but attrs is a common dependency of many Python packages. We do have a hard dependency on its internals indirectly because we pickle the license detection index and there are attrs-based objects in this. Therefore we cannot be subject to any version update that we do not control. But we cannot either pin our version exactly or else we make the life of our users miserable as we will break their deps tree. There is not much to discuss with upstream as they even non-API breaking changes would impact us (and they did and were a major source of churn and user issues late last year)

* [spdx-tools](https://pypi.org/project/spdx-tools/)
* lots of PRs/issues
* licenses.json may be embedded data copy
* otherwise fine
* Philippe's comment: I used to be maintain this and we may drop using this in the future as there is too much complexity attached to its RDF internals.

* [typecode](https://pypi.org/project/typecode/): created by nexB
* vendors [pygments](https://pypi.org/project/pygments/)
* suggest discussing issues upstream
* suggest devendoring
* Philippe's comment: we vendored pygments for the same reason we vendored attrs. This is a widely used library and we depend on pygments lexers stability and they change quite often and break some of our features.

* [typecode-libmagic](https://pypi.org/project/typecode-libmagic/)
* tiny wrapper around libmagic C library
* scancode-plugins has vendored deps and prebuilt files so not packageable
* suggest allowing [python-magic](https://pypi.org/project/python-magic/) as already in Debian
* Philippe's comments: we have code to point to the system-provided libmagic. We are however highly dependent of the stability of the magic database. Note that we call libmagic from https://github.com/nexB/typecode/blob/main/src/typecode/magic2.py which is a patched and updated fork of python-magic upstream and we have discussed with upstream to merge back this, assuming we can agree on how to discover libmagic and the magic db.

* [urlpy](https://pypi.org/project/urlpy/)
* nexB fork of [url](https://pypi.org/project/url/)
* fork reasons not well explained
* upstream is better maintained
* suggest switching
* upstream lacks wheels
* suggest asking them about that
* Philippe's comment: upstream at https://github.com/seomoz/url-py has not been updated in 3+ years and is abandon-war; It further switched to a native C/C++-only Python extension and we decided to fork and maintain the last pure Python version.

As actions are taken, please update this issue description so we can track the current status of each item.

PS: @jelmer has also [looked](https://wiki.debian.org/JelmerVernooij/scancode) at packaging part of scancode for Debian and there are some notes in that analysis about what each module is for and which might be useful for using scancode from detectors of copyrights, license grants and licenses.