creativecommons / creativecommons/quantifying

Integrating PubMed Central As a DataSource to Improve Quantifying Creative Commons

Open
#233 1 comment 0 reactions 0 assignees View on GitHub
✨ goal: improvement 💻 aspect: code help wanted 🟩 priority: low 🧹 status: ticket work required
Dominant language
Python
Stars
48
Forks
74
PR merge metrics
No merged PRs in 30d

Description

## Problem
Improving Quantifying Creative commons with Medical and life Science records from PubMed.

## Description
This Issue aims to integrate PubMed as a new data source for improving Quantifying Creative Commons project. PubMed provides access to biomedical literature with Creative Commons licensing information, contributing valuable insights into open access scientific publishing trends.

## Additional context
### API Documentation
- **[NCBI E-utilities Overview](https://www.ncbi.nlm.nih.gov/books/NBK25501/)** - Complete guide to Entrez Programming Utilities
- **[ESearch Documentation](https://www.ncbi.nlm.nih.gov/books/NBK25499/#chapter4.ESearch)** - Search parameter reference and examples
- **[EFetch Documentation](https://www.ncbi.nlm.nih.gov/books/NBK25499/#chapter4.EFetch)** - Data retrieval specifications and XML schemas
- **[PubMed Search Field Descriptions](https://pubmed.ncbi.nlm.nih.gov/help/#search-field-descriptions-and-tags)** - Complete field tag reference
- **[E-utilities Usage Guidelines](https://www.ncbi.nlm.nih.gov/books/NBK25497/)** - Rate limits, best practices, and compliance requirements

## API Limitations and Constraints
### Rate Limiting
- **Maximum Rate**: 3 requests per second (0.34-second intervals)
- **Enforcement**: Implemented via `time.sleep(0.34)` between requests
- **Retry Strategy**: 3 retries with exponential backoff for failed requests

### Data Retrieval Limits
- **Per Request**: Maximum 9,999 records per `esearch` call
- **Batch Size**: 200 papers per `efetch` request for optimal performance
- **Total Limit**: Configurable via `--fetch-limit` parameter (default: 5,000)

### PubMed Data Source Information
- **[PubMed Homepage](https://pubmed.ncbi.nlm.nih.gov/)** - Main search interface and database statistics
- **[PubMed Data Distribution](https://www.nlm.nih.gov/databases/download/pubmed_medline.html)** - Bulk data access and licensing information
- **[MEDLINE/PubMed Data Element Descriptions](https://www.nlm.nih.gov/bsd/mms/medlineelements.html)** - Complete metadata field specifications
- **[PubMed Central (PMC)](https://www.ncbi.nlm.nih.gov/pmc/)** - Full-text article repository with CC license information
- **[NIH Public Access Policy](https://publicaccess.nih.gov/)** - Mandate requiring open access for NIH-funded research

## Implementation
- [x] I would be interested in implementing this feature.

Contributor guide

Open the contributing guide

Research direction

Start with the NCBI E-utilities Overview, then read the ESearch and EFetch documentation and the PubMed licensing references. Use the --fetch-limit parameter as the stated configuration entry point; done means PubMed records and Creative Commons licensing data are integrated while respecting batch limits, rate limiting, retries, and backoff.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.