NVIDIA-NeMo / NVIDIA-NeMo/Curator
[FEA] Add license detector for code repositories
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.8k
- Forks
- 328
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 30
Description
Sometimes, it is needed to detect the code license of the corpus being curated.
Some code repositories, such as GitHub, provides an API to identify the repository license, but other code repositories, such as Software Heritage, does not provide that capability.
There is a nice tool, scancode-toolkit that could be used for that purpose.
My understanding is that its API support extracting licenses from disk locations, I think that the solution might fake an in-memory filesystem. I would suggest having a look at https://github.com/pytest-dev/pyfakefs for that purpose.
As an optimization, we might identify the files that might contain a license based on a regex pattern, as detailed in Appendix A.3. from StarCoder 2 and The Stack v2: The Next Generation paper.
Hope it helps!
Miguel
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the proposed scancode-toolkit API and pyfakefs for presenting repository contents as a disk location. Compare the optimization idea with the license-file regex in Appendix A.3 of the StarCoder 2 and The Stack v2 paper. Done means detecting licenses for code repositories whose source does not provide a license API.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100