Scrapy
scrapy support for sublime text
Details
Installs
- Total 2K
- Win 1K
- Mac 389
- Linux 311
| Jul 27 | Jul 26 | Jul 25 | Jul 24 | Jul 23 | Jul 22 | Jul 21 | Jul 20 | Jul 19 | Jul 18 | Jul 17 | Jul 16 | Jul 15 | Jul 14 | Jul 13 | Jul 12 | Jul 11 | Jul 10 | Jul 9 | Jul 8 | Jul 7 | Jul 6 | Jul 5 | Jul 4 | Jul 3 | Jul 2 | Jul 1 | Jun 30 | Jun 29 | Jun 28 | Jun 27 | Jun 26 | Jun 25 | Jun 24 | Jun 23 | Jun 22 | Jun 21 | Jun 20 | Jun 19 | Jun 18 | Jun 17 | Jun 16 | Jun 15 | Jun 14 | Jun 13 | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Windows | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| Mac | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| Linux | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 |
Readme
- Source
- raw.githubusercontent.com
Scrapy Snippets for Sublime Text
This package provides handy snippets for working with scrapy, the python tool for making web spiders. It also provides palette commands for parsing request parameters from Postman.
Installation
Clone this package in your Packages/ directory. No Package Control support yet :)
Snippets
Snippets assume your spider has a BASE_URLattribute. This prevents code repetition and makes your code easily portable in case the website changes it's base url.
Snippets for the most common requests are provided:
- Get: for sending a request with url request parameters, like mysite.com?parameter=value
- Post: for X-www-Form-Urlencoded requests
- Json: for POST requests with application/json content-type
- A 'redirect' method, useful when you need to send a GET request without parameters.
- A 'last' method that opens up the received response in the browser and starts an ipdb shell, useful for debugging the response or playing around with xpaths.
- A asp parameter extractor, for easily getting __VIEWSTATE, __VIEWSTATEGENERATOR and __EVENTVALIDATION from a response.
Commands
Format from Postman
If you're using Postman to inspect your requests to a site, you can copy & paste the parameters in 'bulk-edit' mode, and paste them into your spider. Then, select all the lines you just pasted and from the command palette execute Scrapy: Format from Postman. The parameters will be formatted as python's dictionary keys and values.
Instance Item
Copy an item class definition in your spider. Then, select all the class definition lines and execute Scrapy: Yield Item. The definition will be formatted into a yield ClassInstance() statement.