aboutcode-org / aboutcode-org/scancode-toolkit

Collect all GenAI licenses for correct detection

Open
#4,567 4 comments 0 reactions 1 assignee Assigned to @Godzilaa View on GitHub
Dominant language
Python
Stars
2.6k
Forks
791
Avg merge
1d 12h
Merged PRs (30d)
5

Description

We need to ensure that LLMs have a clear license. This is messy out there with a lot of new, weird, proprietary licenses.

There is something that would be super useful wrt. AI: find all these licenses!

The process would be:
- [ ] Collect the metadata files from all Hugging Face GenAi repos
- [ ] Scan GenAi metadata files with scancode
- [ ] Review GenAi metadata file scan results for correctness
- [ ] Create new licenses, and new license rules used @ HuggingFace

When cloning all the repos @ Hugging Face, these should shallow clones, without Git LFS and without xet large files, and possibly partial clone to focus only on a few files: model cards and licenses, and related metadata files. See for ideas:
- https://github.com/aboutcode-org/federatedcode/issues/82

Running ScanCode license detection on that and review the results to find all the funky new AI-licenses and notices may be best done in ScanCode.io for efficiency, but the licenses are ultimately created in ScanCode toolkit

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.