postmanlabs / postmanlabs/postman-code-generators
New Code Generator - Python Scrapy
Nobody has claimed this yet.
- Dominant language
- JavaScript
- Stars
- 1k
- Forks
- 381
- PR merge metrics
- No merged PRs in 30d
Description
Is your feature request related to a problem? Please describe.
Postman has a codegen for Python requests using the de-facto requests module. Personally I use postman for larger systems designed for scraping websites, and use a common framework called scrapy. This framework builds on top of many python libraries and I often find myself translating from requests or the app itself into scrapy requests (due to syntax differences and custom requests objects). I would find it helpful to have a codegen specifically for scrapy just as there is one for Python's httpclient and requests.
Describe the solution you'd like
Code output in scrapy. using scrapy.Reqeust, scrapy.FormRequest, etc..
Additional context
https://docs.scrapy.org/en/latest/topics/request-response.html
Postman is a great tool for web scraping, may as well have official support for the most popular* framework
* No numerical evidence to support its the most popular framework, purely my opinion.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the existing Python requests and httpclient generators, then read the linked Scrapy request-response documentation. Done means the project has a Scrapy code generator that produces the requested Scrapy request types and integrates with the repository's generator conventions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100