ctrl+shift+p filters: :st2 :st3 :win :osx :linux
Browse

Scrapy

scrapy support for sublime text

Details

Installs

  • Total 2K
  • Win 1K
  • Mac 389
  • Linux 310
Jun 15 Jun 14 Jun 13 Jun 12 Jun 11 Jun 10 Jun 9 Jun 8 Jun 7 Jun 6 Jun 5 Jun 4 Jun 3 Jun 2 Jun 1 May 31 May 30 May 29 May 28 May 27 May 26 May 25 May 24 May 23 May 22 May 21 May 20 May 19 May 18 May 17 May 16 May 15 May 14 May 13 May 12 May 11 May 10 May 9 May 8 May 7 May 6 May 5 May 4 May 3 May 2
Windows 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mac 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Linux 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

Readme

Source
raw.​githubusercontent.​com

Scrapy Snippets for Sublime Text

This package provides handy snippets for working with scrapy, the python tool for making web spiders. It also provides palette commands for parsing request parameters from Postman.

Installation

Clone this package in your Packages/ directory. No Package Control support yet :)

Snippets

Snippets assume your spider has a BASE_URLattribute. This prevents code repetition and makes your code easily portable in case the website changes it's base url.

Snippets for the most common requests are provided: - Get: for sending a request with url request parameters, like mysite.com?parameter=value - Post: for X-www-Form-Urlencoded requests - Json: for POST requests with application/json content-type - A 'redirect' method, useful when you need to send a GET request without parameters. - A 'last' method that opens up the received response in the browser and starts an ipdb shell, useful for debugging the response or playing around with xpaths. - A asp parameter extractor, for easily getting __VIEWSTATE, __VIEWSTATEGENERATOR and __EVENTVALIDATION from a response.

Commands

Format from Postman

If you're using Postman to inspect your requests to a site, you can copy & paste the parameters in 'bulk-edit' mode, and paste them into your spider. Then, select all the lines you just pasted and from the command palette execute Scrapy: Format from Postman. The parameters will be formatted as python's dictionary keys and values.

Instance Item

Copy an item class definition in your spider. Then, select all the class definition lines and execute Scrapy: Yield Item. The definition will be formatted into a yield ClassInstance() statement.