biopython / biopython/biopython
urllib.error.HTTPError: HTTP Error 400: Bad Request with Entrez inside loop
- Dominant language
- Python
- Stars
- 5.2k
- Forks
- 1.9k
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 11
Description
### Setup
Hi! I'm having issue with requests with Entrez. I can't understand this behaviour.
```python
import sys; print(sys.version)
3.10.9 (main, Jan 11 2023, 09:18:20) [Clang 14.0.6 ]
import platform; print(platform.python_implementation()); print(platform.platform())
CPython
macOS-10.14.6-x86_64-i386-64bit
import Bio; print(Bio.__version__)
1.80
```
### Expected behaviour
If i'm running outside ouf my loop my request (example) :
```
>>> handle = Entrez.efetch(db="bioproject", retmode="xml", id=305084)
>>> bio_file = handle.read()
>>> soup = BS.BeautifulSoup(bio_file, 'xml')
>>> soup
bp0
bp0
Bacteria
Great Artesian Basin gas bore wells Genome sequencing and assembly
In an effort to discover novel bacterial species or novel ecotypes with potential bioremediation or biotechnological applications, bacteria were isolated from water/sediment samples taken from gas producing bore wells, some involved in coal seam gas (CSG) extraction activities. Genome sequencing of isolates using the WGS sequencing approach was conducted in an effort to elucidate functional profiles of novel isolates or identify novel functional properties of new ecotypes. This genome sequencing project is part of a larger research effort to better understand the involvement of microbial communities in the production of gas from these bore wells.
2016-02-01T00:00:00Z
yes
AT864
AU359
AU374
AU375
AVP41
AVP42
AVP43
Bacteria
eBacteria
C
eMesophilic
eAquatic
Bharat K. C. Patel
Water sampled from gas producing bore wells of the Surat Basin (GAB) of southern Queensland including bores and produced water treatment ponds from coal seam gas (CSG) extraction plant in the same region.
genome sequencing and assembly
Genome sequencing and assembly
Griffith University
public
>>>
```
Then i can parse the result to predict that there is no publication attached to this projectID number
### Actual behaviour
But, inside my function (i have 306 projectID in my list) i just have an error..
this is my function :
```
#/usr/local/bin/python3
from Bio import Entrez
import bs4 as BS
import lxml
import ipdb
import pandas as pd
import time
Entrez.email = "XXXXXX"
Entrez.api_key ="XXXXXXXX"
def get_publication_infos(bioproject_ids):
#return a dict with information about publication related to assembly through bioproject id
dict_info_journal = dict()
list_odd_blank_assembly = list()
for bio_id in bioproject_ids:
#print(bio_id)
print('Entrez.efetch(db="bioproject", retmode="xml", id=bio_id)' +bio_id)
handle = Entrez.efetch(db="bioproject", retmode="xml", id=bio_id)
bio_file = handle.read()
soup = BS.BeautifulSoup(bio_file, 'xml')
time.sleep(5)
if soup.find_all('Publication') == list():
print(bio_id)
list_odd_blank_assembly.append(bio_id)
continue
else:
if len(soup.find_all('Title')) < 2:
print(bio_id)
list_odd_blank_assembly.append(bio_id)
continue
else:
dict_info_journal[bio_id] = dict()
dict_info_journal[bio_id]['Title'] = soup.find_all('Title')[1].string
dict_info_journal[bio_id]['Journal'] = soup.JournalTitle.string
dict_info_journal[bio_id]['Author'] = soup.Last.string+" et al."
dict_info_journal[bio_id]['Year'] = soup.Year.string
dict_info_journal[bio_id]['Pubmed'] = "https://pubmed.ncbi.nlm.nih.gov/"+soup.find("Publication")['id']
handle.close()
return(dict_info_journal,list_odd_blank_assembly)
(dict_info_journal, list_odd_blank_assembly) = get_publication_infos(bioproject_ids)
and output :
Entrez.efetch(db="bioproject", retmode="xml", id=bio_id)828371
828371
Entrez.efetch(db="bioproject", retmode="xml", id=bio_id)605137
605137
Entrez.efetch(db="bioproject", retmode="xml", id=bio_id)911140
911140
Entrez.efetch(db="bioproject", retmode="xml", id=bio_id)696059
696059
Entrez.efetch(db="bioproject", retmode="xml", id=bio_id)310789
310789
Entrez.efetch(db="bioproject", retmode="xml", id=bio_id)305084
urllib.error.HTTPError: HTTP Error 400: Bad Request
```
then i can't understand why the result is not the same in both situation ? i used sleep between my request, didn't change anything..
if anyone has any idea to help me! thanks :)
for the record, i'm trying to filter assembly not related to a publication with projectid number.
Contributor guide
Assessment
This issue has not been assessed yet.