Frequently Asked Questions¶
How does Scrapy compare to BeautifulSoup or lxml?¶
Scrapy provides a built-in mechanism for extracting data (called selectors) but you can easily use BeautifulSoup (or lxml) instead, if you feel more comfortable working with them. After all, they’re just parsing libraries which can be imported and used from any Python code.
What Python versions does Scrapy support?¶
Scrapy is supported under Python 2.7 only. Python 2.6 support was dropped starting at Scrapy 0.20.
Does Scrapy work with Python 3?¶
No, but there are plans to support Python 3.3+. At the moment, Scrapy works with Python 2.7.
Did Scrapy “steal” X from Django?¶
Probably, but we don’t like that word. We think Django is a great open source project and an example to follow, so we’ve used it as an inspiration for Scrapy.
We believe that, if something is already done well, there’s no need to reinvent it. This concept, besides being one of the foundations for open source and free software, not only applies to software but also to documentation, procedures, policies, etc. So, instead of going through each problem ourselves, we choose to copy ideas from those projects that have already solved them properly, and focus on the real problems we need to solve.
We’d be proud if Scrapy serves as an inspiration for other projects. Feel free to steal from us!
Does Scrapy work with HTTP proxies?¶
Yes. Support for HTTP proxies is provided (since Scrapy 0.8) through the HTTP Proxy downloader middleware. See HttpProxyMiddleware.
How can I scrape an item with attributes in different pages?¶
Scrapy crashes with: ImportError: No module named win32api¶
How can I simulate a user login in my spider?¶
Does Scrapy crawl in breadth-first or depth-first order?¶
By default, Scrapy uses a LIFO queue for storing pending requests, which basically means that it crawls in DFO order. This order is more convenient in most cases. If you do want to crawl in true BFO order, you can do it by setting the following settings:
DEPTH_PRIORITY = 1 SCHEDULER_DISK_QUEUE = 'scrapy.squeue.PickleFifoDiskQueue' SCHEDULER_MEMORY_QUEUE = 'scrapy.squeue.FifoMemoryQueue'
My Scrapy crawler has memory leaks. What can I do?¶
Also, Python has a builtin memory leak issue which is described in Leaks without leaks.
How can I make Scrapy consume less memory?¶
See previous question.
Why does Scrapy download pages in English instead of my native language?¶
Can I run a spider without creating a project?¶
Yes. You can use the runspider command. For example, if you have a spider written in a my_spider.py file you can run it with:
scrapy runspider my_spider.py
See runspider command for more info.
I get “Filtered offsite request” messages. How can I fix them?¶
Those messages (logged with DEBUG level) don’t necessarily mean there is a problem, so you may not need to fix them.
Those message are thrown by the Offsite Spider Middleware, which is a spider middleware (enabled by default) whose purpose is to filter out requests to domains outside the ones covered by the spider.
For more info see: OffsiteMiddleware.
Can I use JSON for large exports?¶
Can I return (Twisted) deferreds from signal handlers?¶
Some signals support returning deferreds from their handlers, others don’t. See the Built-in signals reference to know which ones.
What does the response status code 999 means?¶
999 is a custom reponse status code used by Yahoo sites to throttle requests. Try slowing down the crawling speed by using a download delay of 2 (or higher) in your spider:
class MySpider(CrawlSpider): name = 'myspider' download_delay = 2 # [ ... rest of the spider code ... ]
Or by setting a global download delay in your project with the DOWNLOAD_DELAY setting.
Can I call pdb.set_trace() from my spiders to debug them?¶
Yes, but you can also use the Scrapy shell which allows you too quickly analyze (and even modify) the response being processed by your spider, which is, quite often, more useful than plain old pdb.set_trace().
For more info see Invoking the shell from spiders to inspect responses.
Simplest way to dump all my scraped items into a JSON/CSV/XML file?¶
To dump into a JSON file:
scrapy crawl myspider -o items.json -t json
To dump into a CSV file:
scrapy crawl myspider -o items.csv -t csv
To dump into a XML file:
scrapy crawl myspider -o items.xml -t xml
For more information see Feed exports
What’s this huge cryptic __VIEWSTATE parameter used in some forms?¶
What’s the best way to parse big XML/CSV data feeds?¶
Parsing big feeds with XPath selectors can be problematic since they need to build the DOM of the entire feed in memory, and this can be quite slow and consume a lot of memory.
In order to avoid parsing all the entire feed at once in memory, you can use the functions xmliter and csviter from scrapy.utils.iterators module. In fact, this is what the feed spiders (see Spiders) use under the cover.
How can I instruct a spider to stop itself?¶
Should I use spider arguments or settings to configure my spider?¶
Both spider arguments and settings can be used to configure your spider. There is no strict rule that mandates to use one or the other, but settings are more suited for parameters that, once set, don’t change much, while spider arguments are meant to change more often, even on each spider run and sometimes are required for the spider to run at all (for example, to set the start url of a spider).
To illustrate with an example, assuming you have a spider that needs to log into a site to scrape data, and you only want to scrape data from a certain section of the site (which varies each time). In that case, the credentials to log in would be settings, while the url of the section to scrape would be a spider argument.
I’m scraping a XML document and my XPath selector doesn’t return any items¶
You may need to remove namespaces. See Removing namespaces.
I’m getting an error: “cannot import name crawler”¶
This is caused by Scrapy changes due to the singletons removal. The error is most likely raised by a module (extension, middleware, pipeline or spider) in your Scrapy project that imports crawler from scrapy.project. For example:
from scrapy.project import crawler class SomeExtension(object): def __init__(self): self.crawler = crawler # ...
This way to access the crawler object is deprecated, the code should be ported to use from_crawler class method, for example:
class SomeExtension(object): @classmethod def from_crawler(cls, crawler): o = cls() o.crawler = crawler return o
Scrapy command line tool has some backwards compatibility in place to support the old import mechanism (with a deprecation warning), but this mechanism may not work if you use Scrapy differently (for example, as a library).