Annotate pre-training datasets for copyright

Task framed within the WP3.4 Regulatory Compliance for detecting licenses and copyright-related content in text pre-training datasets, mainly coming from OCR processes, and for filtering out restrictive or copyrighted documents based on that detection.

There is an existing module developed and tested by the BSC, mainly consisting of a set of rules and heuristics for Spanish and English. The characteristics and limitations will be explained in detailed in this task, along with the repository with the code and test data which will be linked as well. The module originated from the necessity of filtering the FinePDFs dataset, which contains many OCR-based texts with copyright flags.

The expected outcome of this task is:

• Publish copyright detection module on OELLM GitHub.
• Apply copyright detection module to FinePDFs.
• Analyze annotations and data retention in FInePDFS after filtering copyright.
• Integrate filtered FinePDFs in the WP3 Training Catalogue.
• Iterate over the copyright detection module.