jackfrued / jackfrued/Python-100-Days
Scrapy问题
Nobody has claimed this yet.
- Dominant language
- Jupyter Notebook
- Stars
- 187k
- Forks
- 55.8k
- PR merge metrics
- No merged PRs in 30d
Description
在scrapy基础那章节，用scrapy框架进行爬虫时， 请求返回
“>>>response
<200 https://book.douban.com/top250>
response.text
\x15l�QT�ȈԤ\x1e\x00�\x12l\��P�\x1e��#t����������Q�1\x10\x0b$Vo#�e�=\x1c\x01�Q\x0c�\x01���C\u07fc_�Y�\x0fin�FUa�C�,\x9bm���an\x1e��х�[�u�N���t%�}��\n\x08����B\n��?�\x02\x0b^���...“
是如上面所示的结果，尝试过更改编码都不对，对应的headers全部做了伪装；
用该框架爬取百度百科时返回的网页源码是正确的；用request+BeautifulSoup请求解析豆瓣的网页，返回的源码也是正确的；
由于可以正常请求网页，故出现上述scrapy爬虫问题是怎么导致的，应该怎么改动？谢谢
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files or tests are identified. Start by reproducing the Scrapy request to https://book.douban.com/top250 and inspect the response headers and body encoding or compression; done means explaining why the response differs from requests and documenting a verified configuration or request change.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- web-dev
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100