creativecommons / creativecommons/quantifying
Integrating PubMed Central As a DataSource to Improve Quantifying Creative Commons
- Langage dominant
- Python
- Étoiles
- 48
- Forks
- 74
- Métriques de merge des PR
- Aucune PR mergée en 30 j
Description
## Problem
Improving Quantifying Creative commons with Medical and life Science records from PubMed.
## Description
This Issue aims to integrate PubMed as a new data source for improving Quantifying Creative Commons project. PubMed provides access to biomedical literature with Creative Commons licensing information, contributing valuable insights into open access scientific publishing trends.
## Additional context
### API Documentation
- **[NCBI E-utilities Overview](https://www.ncbi.nlm.nih.gov/books/NBK25501/)** - Complete guide to Entrez Programming Utilities
- **[ESearch Documentation](https://www.ncbi.nlm.nih.gov/books/NBK25499/#chapter4.ESearch)** - Search parameter reference and examples
- **[EFetch Documentation](https://www.ncbi.nlm.nih.gov/books/NBK25499/#chapter4.EFetch)** - Data retrieval specifications and XML schemas
- **[PubMed Search Field Descriptions](https://pubmed.ncbi.nlm.nih.gov/help/#search-field-descriptions-and-tags)** - Complete field tag reference
- **[E-utilities Usage Guidelines](https://www.ncbi.nlm.nih.gov/books/NBK25497/)** - Rate limits, best practices, and compliance requirements
## API Limitations and Constraints
### Rate Limiting
- **Maximum Rate**: 3 requests per second (0.34-second intervals)
- **Enforcement**: Implemented via `time.sleep(0.34)` between requests
- **Retry Strategy**: 3 retries with exponential backoff for failed requests
### Data Retrieval Limits
- **Per Request**: Maximum 9,999 records per `esearch` call
- **Batch Size**: 200 papers per `efetch` request for optimal performance
- **Total Limit**: Configurable via `--fetch-limit` parameter (default: 5,000)
### PubMed Data Source Information
- **[PubMed Homepage](https://pubmed.ncbi.nlm.nih.gov/)** - Main search interface and database statistics
- **[PubMed Data Distribution](https://www.nlm.nih.gov/databases/download/pubmed_medline.html)** - Bulk data access and licensing information
- **[MEDLINE/PubMed Data Element Descriptions](https://www.nlm.nih.gov/bsd/mms/medlineelements.html)** - Complete metadata field specifications
- **[PubMed Central (PMC)](https://www.ncbi.nlm.nih.gov/pmc/)** - Full-text article repository with CC license information
- **[NIH Public Access Policy](https://publicaccess.nih.gov/)** - Mandate requiring open access for NIH-funded research
## Implementation
- [x] I would be interested in implementing this feature.
Guide de contribution
Ouvrir le guide de contribution
Piste de recherche
Commencez par la présentation générale de NCBI E-utilities, puis lisez la documentation de ESearch et EFetch ainsi que les références de licence de PubMed. Utilisez le paramètre --fetch-limit comme point d’entrée de configuration indiqué ; le travail est considéré comme terminé lorsque les enregistrements PubMed et les données de licence Creative Commons sont intégrés tout en respectant les limites de lot, la limitation du débit, les nouvelles tentatives et le backoff.
Rédigé par le modèle d'indexation à partir du texte de l'issue.
Évaluation
- Stack technique
- python
- Domaine
- data
- Type d'issue
- Fonctionnalité
- Difficulté
- 5/5
- Temps estimé
- Plus d'une semaine
- Activité
- À l'abandon
- Clarté
- Plutôt claire
- Accessibilité débutants
- 35/100