postmanlabs / postmanlabs/postman-code-generators

New Code Generator - Python Scrapy

Open
#533 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

codegen feature new-language-support
Dominant language
JavaScript
Stars
1k
Forks
381
PR merge metrics
No merged PRs in 30d

Description

Is your feature request related to a problem? Please describe.
Postman has a codegen for Python requests using the de-facto requests module. Personally I use postman for larger systems designed for scraping websites, and use a common framework called scrapy. This framework builds on top of many python libraries and I often find myself translating from requests or the app itself into scrapy requests (due to syntax differences and custom requests objects). I would find it helpful to have a codegen specifically for scrapy just as there is one for Python's httpclient and requests.

Describe the solution you'd like
Code output in scrapy. using scrapy.Reqeust, scrapy.FormRequest, etc..

Additional context
https://docs.scrapy.org/en/latest/topics/request-response.html
Postman is a great tool for web scraping, may as well have official support for the most popular* framework

* No numerical evidence to support its the most popular framework, purely my opinion.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the existing Python requests and httpclient generators, then read the linked Scrapy request-response documentation. Done means the project has a Scrapy code generator that produces the requested Scrapy request types and integrates with the repository's generator conventions.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.