creativecommons / creativecommons/quantifying
Integrating PubMed Central As a DataSource to Improve Quantifying Creative Commons
- Vorherrschende Sprache
- Python
- Sterne
- 48
- Forks
- 74
- PR-Merge-Kennzahlen
- Keine gemergten PRs in 30 T.
Beschreibung
## Problem
Improving Quantifying Creative commons with Medical and life Science records from PubMed.
## Description
This Issue aims to integrate PubMed as a new data source for improving Quantifying Creative Commons project. PubMed provides access to biomedical literature with Creative Commons licensing information, contributing valuable insights into open access scientific publishing trends.
## Additional context
### API Documentation
- **[NCBI E-utilities Overview](https://www.ncbi.nlm.nih.gov/books/NBK25501/)** - Complete guide to Entrez Programming Utilities
- **[ESearch Documentation](https://www.ncbi.nlm.nih.gov/books/NBK25499/#chapter4.ESearch)** - Search parameter reference and examples
- **[EFetch Documentation](https://www.ncbi.nlm.nih.gov/books/NBK25499/#chapter4.EFetch)** - Data retrieval specifications and XML schemas
- **[PubMed Search Field Descriptions](https://pubmed.ncbi.nlm.nih.gov/help/#search-field-descriptions-and-tags)** - Complete field tag reference
- **[E-utilities Usage Guidelines](https://www.ncbi.nlm.nih.gov/books/NBK25497/)** - Rate limits, best practices, and compliance requirements
## API Limitations and Constraints
### Rate Limiting
- **Maximum Rate**: 3 requests per second (0.34-second intervals)
- **Enforcement**: Implemented via `time.sleep(0.34)` between requests
- **Retry Strategy**: 3 retries with exponential backoff for failed requests
### Data Retrieval Limits
- **Per Request**: Maximum 9,999 records per `esearch` call
- **Batch Size**: 200 papers per `efetch` request for optimal performance
- **Total Limit**: Configurable via `--fetch-limit` parameter (default: 5,000)
### PubMed Data Source Information
- **[PubMed Homepage](https://pubmed.ncbi.nlm.nih.gov/)** - Main search interface and database statistics
- **[PubMed Data Distribution](https://www.nlm.nih.gov/databases/download/pubmed_medline.html)** - Bulk data access and licensing information
- **[MEDLINE/PubMed Data Element Descriptions](https://www.nlm.nih.gov/bsd/mms/medlineelements.html)** - Complete metadata field specifications
- **[PubMed Central (PMC)](https://www.ncbi.nlm.nih.gov/pmc/)** - Full-text article repository with CC license information
- **[NIH Public Access Policy](https://publicaccess.nih.gov/)** - Mandate requiring open access for NIH-funded research
## Implementation
- [x] I would be interested in implementing this feature.
Beitragsleitfaden
Rechercherichtung
Beginne mit der Übersicht zu den NCBI E-utilities, lies anschließend die Dokumentation zu ESearch und EFetch sowie die Lizenzierungsreferenzen von PubMed. Verwende den Parameter --fetch-limit als angegebenen Konfigurationseinstiegspunkt; als abgeschlossen gilt die Aufgabe, wenn PubMed-Datensätze und Creative-Commons-Lizenzierungsdaten unter Einhaltung von Batch-Limits, Ratenbegrenzung, Wiederholungsversuchen und Backoff integriert sind.
Vom Indexierungsmodell aus dem Issue-Text verfasst.
Bewertung
- Tech-Stack
- python
- Bereich
- data
- Issue-Typ
- Feature
- Schwierigkeit
- 5/5
- Geschätzter Aufwand
- Über eine Woche
- Aktivitätsstatus
- Veraltet
- Klarheit
- Größtenteils klar
- Anfängerfreundlichkeit
- 35/100