# Scrapy Documentation
Source: /index.md
## Scrapy 2.19 documentation
Scrapy is a fast high-level [web crawling](https://en.wikipedia.org/wiki/Web_crawler) and [web scraping](https://en.wikipedia.org/wiki/Web_scraping) framework, used
to crawl websites and extract structured data from their pages. It can be used
for a wide range of purposes, from data mining to monitoring and automated
testing.
### Getting help
Having trouble? We’d like to help!
* Try the FAQ – it’s got answers to some common questions.
* Looking for specific information? Try the genindex or modindex.
* Ask or search questions in [StackOverflow using the scrapy tag](https://stackoverflow.com/tags/scrapy).
* Ask or search questions in the [Scrapy subreddit](https://www.reddit.com/r/scrapy/).
* Search for questions on the archives of the [scrapy-users mailing list](https://groups.google.com/forum/#!forum/scrapy-users).
* Ask a question in the [#scrapy IRC channel](irc://irc.freenode.net/scrapy).
* Report bugs with Scrapy in our [issue tracker](https://github.com/scrapy/scrapy/issues).
* Join the Discord community [Scrapy Discord](https://discord.com/invite/mv3yErfpvq).
### First steps
intro/overview
: Understand what Scrapy is and how it can help you.
intro/install
: Get Scrapy installed on your computer.
intro/tutorial
: Write your first Scrapy project.
intro/examples
: Learn more by playing with a pre-made Scrapy project.
### Basic concepts
topics/concepts
: Get a quick map of Scrapy’s main concepts and when to use each.
topics/commands
: Learn about the command-line tool used to manage your Scrapy project.
topics/spiders
: Write the rules to crawl your websites.
topics/selectors
: Extract the data from web pages using XPath.
topics/items
: Define the data you want to scrape.
topics/loaders
: Populate your items with the extracted data.
topics/shell
: Test your extraction code in an interactive environment.
topics/item-pipeline
: Post-process and store your scraped data.
topics/feed-exports
: Output your scraped data using different formats and storages.
topics/request-response
: Understand the classes used to represent HTTP requests and responses.
topics/cookies
: Send and receive cookies.
topics/link-extractors
: Convenient classes to extract links to follow from pages.
topics/settings
: Learn how to configure Scrapy and see all available settings.
topics/exceptions
: See all available exceptions and their meaning.
### Built-in services
topics/logging
: Learn how to use Python’s built-in logging on Scrapy.
topics/stats
: Collect statistics about your scraping crawler.
topics/telnetconsole
: Inspect a running crawler using a built-in Python console.
### Solving specific problems
faq
: Get answers to most frequently asked questions.
topics/agents
: Learn how to use Scrapy with coding agents efficiently.
topics/debug
: Learn how to debug common problems of your Scrapy spider.
topics/contracts
: Learn how to use contracts for testing your spiders.
topics/practices
: Get familiar with some Scrapy common practices.
topics/security
: Understand the security implications of Scrapy defaults and how to harden
them.
topics/optimize
: Find the bottleneck of your crawls and learn how to address it.
topics/developer-tools
: Learn how to scrape with your browser’s developer tools.
topics/dynamic-content
: Read webpage data that is loaded dynamically.
topics/leaks
: Learn how to find and get rid of memory leaks in your crawler.
topics/media-pipeline
: Download files and/or images associated with your scraped items.
topics/deploy
: Deploying your Scrapy spiders and run them in a remote server.
topics/autothrottle
: Adjust crawl rate dynamically based on load.
topics/benchmarking
: Check how Scrapy performs on your hardware.
topics/jobs
: Learn how to pause and resume crawls for large spiders.
topics/coroutines
: Use the coroutine syntax.
topics/asyncio
: Use `asyncio` and `asyncio`-powered libraries.
### Extending Scrapy
topics/architecture
: Understand the Scrapy architecture.
topics/addons
: Enable and configure third-party extensions.
topics/downloader-middleware
: Customize how pages get requested and downloaded.
topics/spider-middleware
: Customize the input and output of your spiders.
topics/extensions
: Extend Scrapy with your custom functionality
topics/signals
: See all available signals and how to work with them.
topics/scheduler
: Understand the scheduler component.
topics/exporters
: Quickly export your scraped items to a file (XML, CSV, etc).
topics/download-handlers
: Customize how requests are downloaded or add support for new URL schemes.
topics/components
: Learn the common API and some good practices when building custom Scrapy
components.
topics/api
: Use it on extensions and middlewares to extend Scrapy functionality.
### All the rest
news
: See what has changed in recent Scrapy versions.
contributing
: Learn how to contribute to the Scrapy project.
versioning
: Understand Scrapy versioning and API stability.
Source: /intro/overview.md
## Scrapy at a glance
Scrapy (/ˈskreɪpaɪ/) is an application framework for crawling web sites and extracting
structured data which can be used for a wide range of useful applications, like
data mining, information processing or historical archival.
Even though Scrapy was originally designed for [web scraping](https://en.wikipedia.org/wiki/Web_scraping), it can also be
used to extract data using APIs (such as [Amazon Associates Web Services](https://affiliate-program.amazon.com/welcome/ecs)) or
as a general purpose web crawler.
### Walk-through of an example spider
In order to show you what Scrapy brings to the table, we’ll walk you through an
example of a Scrapy Spider using the simplest way to run a spider.
Here’s the code for a spider that scrapes famous quotes from website
[https://quotes.toscrape.com](https://quotes.toscrape.com), following the pagination:
```python
import scrapy
class QuotesSpider(scrapy.Spider):
name = "quotes"
start_urls = [
"https://quotes.toscrape.com/tag/humor/",
]
def parse(self, response):
for quote in response.css("div.quote"):
yield {
"author": quote.xpath("span/small/text()").get(),
"text": quote.css("span.text::text").get(),
}
next_page = response.css('li.next a::attr("href")').get()
if next_page is not None:
yield response.follow(next_page, self.parse)
```
Put this in a text file, name it something like `quotes_spider.py`
and run the spider using the `runspider` command:
```
scrapy runspider quotes_spider.py -o quotes.jsonl
```
When this finishes you will have in the `quotes.jsonl` file a list of the
quotes in JSON Lines format, containing the text and author, which will look like this:
```
{"author": "Jane Austen", "text": "\u201cThe person, be it gentleman or lady, who has not pleasure in a good novel, must be intolerably stupid.\u201d"}
{"author": "Steve Martin", "text": "\u201cA day without sunshine is like, you know, night.\u201d"}
{"author": "Garrison Keillor", "text": "\u201cAnyone who thinks sitting in church can make you a Christian must also think that sitting in a garage can make you a car.\u201d"}
...
```
#### What just happened?
When you ran the command `scrapy runspider quotes_spider.py`, Scrapy looked for a
Spider definition inside it and ran it through its crawler engine.
The crawl started by making requests to the URLs defined in the `start_urls`
attribute (in this case, only the URL for quotes in the *humor* category)
and called the default callback method `parse`, passing the response object as
an argument. In the `parse` callback, we loop through the quote elements
using a CSS Selector, yield a Python dict with the extracted quote text and author,
look for a link to the next page and schedule another request using the same
`parse` method as callback.
Here you will notice one of the main advantages of Scrapy: requests are
scheduled and processed asynchronously. This
means that Scrapy doesn’t need to wait for a request to be finished and
processed, it can send another request or do other things in the meantime. This
also means that other requests can keep going even if a request fails or an
error happens while handling it.
While this enables you to do very fast crawls (sending multiple concurrent
requests at the same time, in a fault-tolerant way) Scrapy also gives you
control over the politeness of the crawl through a few settings. You can do things like setting a download delay between
each request, limiting the amount of concurrent requests per domain, and
even using an auto-throttling extension that tries
to figure these settings out automatically.
> [!NOTE]
> This is using feed exports to generate the
> JSON Lines file, you can easily change the export format (XML or CSV, for
> example) or the storage backend (FTP or [Amazon S3](https://aws.amazon.com/s3/), for example). You can
> also write an item pipeline to store the
> items in a database.
### What else?
You’ve seen how to extract and store items from a website using Scrapy, but
this is just the surface. Scrapy provides a lot of powerful features for making
scraping easy and efficient, such as:
* Built-in support for selecting and extracting data
from HTML/XML sources using extended CSS selectors and XPath expressions,
with helper methods for extraction using regular expressions.
* An interactive shell console (IPython aware) for trying
out the CSS and XPath expressions to scrape data, which is very useful when writing or
debugging your spiders.
* Built-in support for generating feed exports in
multiple formats (JSON, CSV, XML) and storing them in multiple backends (FTP,
S3, local filesystem)
* Robust encoding support and auto-detection, for dealing with foreign,
non-standard and broken encoding declarations.
* Strong extensibility support, allowing you to plug
in your own functionality using signals and a
well-defined API (middlewares, extensions, and
pipelines).
* A wide range of built-in extensions and middlewares for handling:
- cookies and session handling
- HTTP features like compression, authentication, caching
- user-agent spoofing
- robots.txt
- crawl depth restriction
- and more
* Several ways to execute code inside a Scrapy process, to introspect and debug it.
* An official plugin for coding agents.
* Plus other goodies like reusable spiders to crawl sites from [Sitemaps](https://www.sitemaps.org/index.html) and
XML/CSV feeds, a media pipeline for automatically downloading images (or any other media) associated with the scraped
items, a caching DNS resolver, and much more!
### What’s next?
The next steps for you are to install Scrapy,
follow through the tutorial to learn how to create
a full-blown Scrapy project and [join the community](https://www.scrapy.org/community). Thanks for your
interest!
Source: /intro/install.md
## Installation guide
### Supported Python versions
Scrapy requires Python 3.10+, either the CPython implementation (default) or
the PyPy implementation (see python:implementations).
### Installing Scrapy
If you’re using [Anaconda](https://www.anaconda.com/docs/main) or [Miniconda](https://docs.conda.io/projects/conda/en/latest/user-guide/install/index.html), you can install the package from
the [conda-forge](https://conda-forge.org/) channel, which has up-to-date packages for Linux, Windows
and macOS.
To install Scrapy using `conda`, run:
```
conda install -c conda-forge scrapy
```
Alternatively, if you’re already familiar with installation of Python packages,
you can install Scrapy and its dependencies from PyPI with:
```
pip install Scrapy
```
We strongly recommend that you install Scrapy in a dedicated virtualenv,
to avoid conflicting with your system packages.
Note that sometimes this may require solving compilation issues for some Scrapy
dependencies depending on your operating system, so be sure to check the
intro-install-platform-notes.
For more detailed and platform-specific instructions, as well as
troubleshooting information, read on.
#### Things that are good to know
Scrapy is written in pure Python and depends on a few key Python packages (among others):
* [lxml](https://lxml.de/index.html), an efficient XML and HTML parser
* [parsel](https://pypi.org/project/parsel/), an HTML/XML data extraction library written on top of lxml,
* [w3lib](https://pypi.org/project/w3lib/), a multi-purpose helper for dealing with URLs and web page encodings
* [twisted](https://twisted.org/), an asynchronous networking framework
* [cryptography](https://cryptography.io/en/latest/) and [pyOpenSSL](https://pypi.org/project/pyOpenSSL/), to deal with various network-level security needs
Some of these packages themselves depend on non-Python packages
that might require additional installation steps depending on your platform.
Please check platform-specific guides below.
In case of any trouble related to these dependencies,
please refer to their respective installation instructions:
* [lxml installation](https://lxml.de/installation.html)
* cryptography installation
#### Using a virtual environment (recommended)
TL;DR: We recommend installing Scrapy inside a virtual environment
on all platforms.
Python packages can be installed either globally (a.k.a system wide),
or in user-space. We do not recommend installing Scrapy system wide.
Instead, we recommend that you install Scrapy within a so-called
“virtual environment” (`venv`).
Virtual environments allow you to not conflict with already-installed Python
system packages (which could break some of your system tools and scripts),
and still install packages normally with `pip` (without `sudo` and the likes).
See tut-venv on how to create your virtual environment.
Once you have created a virtual environment, you can install Scrapy inside it with `pip`,
just like any other Python package.
(See platform-specific guides
below for non-Python dependencies that you may need to install beforehand).
### Optional extras
Scrapy provides optional extras
that install additional dependencies to enable specific features. To install
Scrapy with one or more extras, list them in square brackets:
```console
pip install scrapy[s3,images]
```
The following extras are available:
| Extra | Provides |
|-----------------|-----------------------------------------------------------------------------------|
| `bpython` | bpython shell |
| `color` | `LOG_COLOR` |
| `gcs` | Google Cloud Storage for
from twisted.internet._sslverify import _setAcceptableProtocols
File "[…]/site-packages/twisted/internet/_sslverify.py", line 38, in
TLSVersion.TLSv1_1: SSL.OP_NO_TLSv1_1,
AttributeError: 'module' object has no attribute 'OP_NO_TLSv1_1'
```
The reason you get this exception is that your system or virtual environment
has a version of pyOpenSSL that your version of Twisted does not support.
To install a version of pyOpenSSL that your version of Twisted supports,
reinstall Twisted with the `tls` extra option:
```
pip install twisted[tls]
```
For details, see [Issue #2473](https://github.com/scrapy/scrapy/issues/2473).
Source: /intro/tutorial.md
## Scrapy Tutorial
In this tutorial, we’ll assume that Scrapy is already installed on your system.
If that’s not the case, see intro-install.
We are going to scrape [quotes.toscrape.com](https://quotes.toscrape.com/), a website
that lists quotes from famous authors.
This tutorial will walk you through these tasks:
1. Creating a new Scrapy project
2. Writing a spider to crawl a site and extract data
3. Exporting the scraped data using the command line
4. Changing spider to recursively follow links
5. Using spider arguments
Scrapy is written in [Python](https://www.python.org/). The more you learn about Python, the more you
can get out of Scrapy.
If you’re already familiar with other languages and want to learn Python quickly, the
[Python Tutorial](https://docs.python.org/3/tutorial) is a good resource.
If you’re new to programming and want to start with Python, the following books
may be useful to you:
* [Automate the Boring Stuff With Python](https://automatetheboringstuff.com/)
* [How To Think Like a Computer Scientist](http://openbookproject.net/thinkcs/python/english3e/)
* [Learn Python 3 The Hard Way](https://learnpythonthehardway.org/python3/)
You can also take a look at [this list of Python resources for non-programmers](https://wiki.python.org/moin/BeginnersGuide/NonProgrammers),
as well as the [suggested resources in the learnpython-subreddit](https://www.reddit.com/r/learnpython/wiki/index#wiki_new_to_python.3F).
### Creating a project
Before you start scraping, you will have to set up a new Scrapy project. Enter a
directory where you’d like to store your code and run:
```
scrapy startproject tutorial
```
This will create a `tutorial` directory with the following contents:
```
tutorial/
scrapy.cfg # deploy configuration file
tutorial/ # project's Python module, you'll import your code from here
__init__.py
items.py # project items definition file
middlewares.py # project middlewares file
pipelines.py # project pipelines file
settings.py # project settings file
spiders/ # a directory where you'll later put your spiders
__init__.py
```
Before crawling anything, open `settings.py` and uncomment the
`USER_AGENT` line to identify yourself, e.g. a project name plus a URL
or an email address. Website owners who take issue with your crawler can then
ask you to adjust it, rather than block it.
### Our first Spider
Spiders are classes that you define and that Scrapy uses to scrape information from a website
(or a group of websites). They must subclass `Spider` and define the initial
requests to be made, and optionally, how to follow links in pages and parse the downloaded
page content to extract data.
This is the code for our first Spider. Save it in a file named
`quotes_spider.py` under the `tutorial/spiders` directory in your project:
```python
from pathlib import Path
import scrapy
class QuotesSpider(scrapy.Spider):
name = "quotes"
async def start(self):
urls = [
"https://quotes.toscrape.com/page/1/",
"https://quotes.toscrape.com/page/2/",
]
for url in urls:
yield scrapy.Request(url=url, callback=self.parse)
def parse(self, response):
page = response.url.split("/")[-2]
filename = f"quotes-{page}.html"
Path(filename).write_bytes(response.body)
self.log(f"Saved file {filename}")
```
As you can see, our Spider subclasses `scrapy.Spider`
and defines some attributes and methods:
* `name`: identifies the Spider. It must be
unique within a project, that is, you can’t set the same name for different
Spiders.
* `start()`: must be an asynchronous generator that
yields requests (and, optionally, items) for the spider to start crawling.
Subsequent requests will be generated successively from these initial
requests.
* `parse()`: a method that will be called to handle
the response downloaded for each of the requests made. The response parameter
is an instance of `TextResponse` that holds
the page content and has further helpful methods to handle it.
The `parse()` method usually parses the response, extracting
the scraped data as dicts and also finding new URLs to
follow and creating new requests (`Request`) from them.
#### How to run our spider
To put our spider to work, go to the project’s top level directory and run:
```
scrapy crawl quotes
```
This command runs the spider named `quotes` that we’ve just added, that
will send some requests for the `quotes.toscrape.com` domain. You will get an output
similar to this:
```
... (omitted for brevity)
2016-12-16 21:24:05 [scrapy.core.engine] INFO: Spider opened
2016-12-16 21:24:05 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2016-12-16 21:24:05 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2016-12-16 21:24:05 [scrapy.core.engine] DEBUG: Crawled (404) (referer: None)
2016-12-16 21:24:05 [scrapy.core.engine] DEBUG: Crawled (200)
[s] item {}
[s] request
[s] spider
[s] Useful shortcuts:
[s] shelp() Shell help (print this help)
[s] fetch(req_or_url) Fetch request (or URL) and update local objects
[s] view(response) View response in a browser
```
Using the shell, you can try selecting elements using [CSS](https://www.w3.org/TR/selectors) with the response
object:
```pycon
>>> response.css("title")
[]
```
The result of running `response.css('title')` is a list-like object called
`SelectorList`, which represents a list of
`Selector` objects that wrap around XML/HTML elements
and allow you to run further queries to refine the selection or extract the
data.
To extract the text from the title above, you can do:
```pycon
>>> response.css("title::text").getall()
['Quotes to Scrape']
```
There are two things to note here: one is that we’ve added `::text` to the
CSS query, to mean we want to select only the text elements directly inside
`` element. If we don’t specify `::text`, we’d get the full title
element, including its tags:
```pycon
>>> response.css("title").getall()
['<title>Quotes to Scrape ']
```
The other thing is that the result of calling `.getall()` is a list: it is
possible that a selector returns more than one result, so we extract them all.
When you know you just want the first result, as in this case, you can do:
```pycon
>>> response.css("title::text").get()
'Quotes to Scrape'
```
As an alternative, you could’ve written:
```pycon
>>> response.css("title::text")[0].get()
'Quotes to Scrape'
```
Accessing an index on a `SelectorList` instance will
raise an `IndexError` exception if there are no results:
```pycon
>>> response.css("noelement")[0].get()
Traceback (most recent call last):
...
IndexError: list index out of range
```
You might want to use `.get()` directly on the
`SelectorList` instance instead, which returns `None`
if there are no results:
```pycon
>>> response.css("noelement").get()
```
There’s a lesson here: for most scraping code, you want it to be resilient to
errors due to things not being found on a page, so that even if some parts fail
to be scraped, you can at least get **some** data.
Besides the `getall()` and
`get()` methods, you can also use
the `re()` method to extract using
regular expressions:
```pycon
>>> response.css("title::text").re(r"Quotes.*")
['Quotes to Scrape']
>>> response.css("title::text").re(r"Q\w+")
['Quotes']
>>> response.css("title::text").re(r"(\w+) to (\w+)")
['Quotes', 'Scrape']
```
In order to find the proper CSS selectors to use, you might find it useful to open
the response page from the shell in your web browser using `view(response)`.
You can use your browser’s developer tools to inspect the HTML and come up
with a selector (see topics-developer-tools).
[Selector Gadget](https://selectorgadget.com/) is also a nice tool to quickly find CSS selector for
visually selected elements, which works in many browsers.
##### XPath: a brief intro
Besides [CSS](https://www.w3.org/TR/selectors), Scrapy selectors also support using [XPath](https://www.w3.org/TR/xpath-10/) expressions:
```pycon
>>> response.xpath("//title")
[]
>>> response.xpath("//title/text()").get()
'Quotes to Scrape'
```
XPath expressions are very powerful, and are the foundation of Scrapy
Selectors. In fact, CSS selectors are converted to XPath under-the-hood. You
can see that if you read the text representation of the selector
objects in the shell closely.
While perhaps not as popular as CSS selectors, XPath expressions offer more
power because besides navigating the structure, it can also look at the
content. Using XPath, you’re able to select things like: *the link
that contains the text “Next Page”*. This makes XPath very fitting to the task
of scraping, and we encourage you to learn XPath even if you already know how to
construct CSS selectors, it will make scraping much easier.
We won’t cover much of XPath here, but you can read more about using XPath
with Scrapy Selectors here. To learn more about XPath, we
recommend [this tutorial to learn XPath through examples](http://zvon.org/comp/r/tut-XPath_1.html), and [this tutorial to learn “how
to think in XPath”](http://plasmasturm.org/log/xpath101/).
##### Extracting quotes and authors
Now that you know a bit about selection and extraction, let’s complete our
spider by writing the code to extract the quotes from the web page.
Each quote in [https://quotes.toscrape.com](https://quotes.toscrape.com) is represented by HTML elements that look
like this:
```html
“The world as we have created it is a process of our
thinking. It cannot be changed without changing our thinking.”
by Albert Einstein
(about)
```
Let’s open up scrapy shell and play a bit to find out how to extract the data
we want:
```
scrapy shell 'https://quotes.toscrape.com'
```
We get a list of selectors for the quote HTML elements with:
```pycon
>>> response.css("div.quote")
[,
,
...]
```
Each of the selectors returned by the query above allows us to run further
queries over their sub-elements. Let’s assign the first selector to a
variable, so that we can run our CSS selectors directly on a particular quote:
```pycon
>>> quote = response.css("div.quote")[0]
```
Now, let’s extract the `text`, `author` and `tags` from that quote
using the `quote` object we just created:
```pycon
>>> text = quote.css("span.text::text").get()
>>> text
'“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”'
>>> author = quote.css("small.author::text").get()
>>> author
'Albert Einstein'
```
Given that the tags are a list of strings, we can use the `.getall()` method
to get all of them:
```pycon
>>> tags = quote.css("div.tags a.tag::text").getall()
>>> tags
['change', 'deep-thoughts', 'thinking', 'world']
```
Having figured out how to extract each bit, we can now iterate over all the
quote elements and put them together into a Python dictionary:
```pycon
>>> for quote in response.css("div.quote"):
... text = quote.css("span.text::text").get()
... author = quote.css("small.author::text").get()
... tags = quote.css("div.tags a.tag::text").getall()
... print(dict(text=text, author=author, tags=tags))
...
{'text': '“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”', 'author': 'Albert Einstein', 'tags': ['change', 'deep-thoughts', 'thinking', 'world']}
{'text': '“It is our choices, Harry, that show what we truly are, far more than our abilities.”', 'author': 'J.K. Rowling', 'tags': ['abilities', 'choices']}
...
```
#### Extracting data in our spider
Let’s get back to our spider. Until now, it hasn’t extracted any data in
particular, just saving the whole HTML page to a local file. Let’s integrate the
extraction logic above into our spider.
A Scrapy spider typically generates many dictionaries containing the data
extracted from the page. To do that, we use the `yield` Python keyword
in the callback, as you can see below:
```python
import scrapy
class QuotesSpider(scrapy.Spider):
name = "quotes"
start_urls = [
"https://quotes.toscrape.com/page/1/",
"https://quotes.toscrape.com/page/2/",
]
def parse(self, response):
for quote in response.css("div.quote"):
yield {
"text": quote.css("span.text::text").get(),
"author": quote.css("small.author::text").get(),
"tags": quote.css("div.tags a.tag::text").getall(),
}
```
To run this spider, exit the scrapy shell by entering:
```
quit()
```
Then, run:
```
scrapy crawl quotes
```
Now, it should output the extracted data with the log:
```
2016-09-19 18:57:19 [scrapy.core.scraper] DEBUG: Scraped from <200 https://quotes.toscrape.com/page/1/>
{'tags': ['life', 'love'], 'author': 'André Gide', 'text': '“It is better to be hated for what you are than to be loved for what you are not.”'}
2016-09-19 18:57:19 [scrapy.core.scraper] DEBUG: Scraped from <200 https://quotes.toscrape.com/page/1/>
{'tags': ['edison', 'failure', 'inspirational', 'paraphrased'], 'author': 'Thomas A. Edison', 'text': "“I have not failed. I've just found 10,000 ways that won't work.”"}
```
### Storing the scraped data
The simplest way to store the scraped data is by using Feed exports, with the following command:
```
scrapy crawl quotes -O quotes.json
```
That will generate a `quotes.json` file containing all scraped items,
serialized in [JSON](https://en.wikipedia.org/wiki/JSON).
The `-O` command-line switch overwrites any existing file; use `-o` instead
to append new content to any existing file. However, appending to a JSON file
makes the file contents invalid JSON. When appending to a file, consider
using a different serialization format, such as [JSON Lines](https://jsonlines.org):
```
scrapy crawl quotes -o quotes.jsonl
```
The [JSON Lines](https://jsonlines.org) format is useful because it’s stream-like, so you can easily
append new records to it. It doesn’t have the same problem as JSON when you run
twice. Also, as each record is a separate line, you can process big files
without having to fit everything in memory, there are tools like [JQ](https://stedolan.github.io/jq) to help
do that at the command-line.
In small projects (like the one in this tutorial), that should be enough.
However, if you want to perform more complex things with the scraped items, you
can write an Item Pipeline. A placeholder file
for Item Pipelines has been set up for you when the project is created, in
`tutorial/pipelines.py`. Though you don’t need to implement any item
pipelines if you just want to store the scraped items.
### Following links
Let’s say, instead of just scraping the stuff from the first two pages
from [https://quotes.toscrape.com](https://quotes.toscrape.com), you want quotes from all the pages in the website.
Now that you know how to extract data from pages, let’s see how to follow links
from them.
The first thing to do is extract the link to the page we want to follow. Examining
our page, we can see there is a link to the next page with the following
markup:
```html
```
We can try extracting it in the shell:
```pycon
>>> response.css('li.next a').get()
'Next → '
```
This gets the anchor element, but we want the attribute `href`. For that,
Scrapy supports a CSS extension that lets you select the attribute contents,
like this:
```pycon
>>> response.css("li.next a::attr(href)").get()
'/page/2/'
```
There is also an `attrib` property available
(see selecting-attributes for more):
```pycon
>>> response.css("li.next a").attrib["href"]
'/page/2/'
```
Now let’s see our spider, modified to recursively follow the link to the next
page, extracting data from it:
```python
import scrapy
class QuotesSpider(scrapy.Spider):
name = "quotes"
start_urls = [
"https://quotes.toscrape.com/page/1/",
]
def parse(self, response):
for quote in response.css("div.quote"):
yield {
"text": quote.css("span.text::text").get(),
"author": quote.css("small.author::text").get(),
"tags": quote.css("div.tags a.tag::text").getall(),
}
next_page = response.css("li.next a::attr(href)").get()
if next_page is not None:
next_page = response.urljoin(next_page)
yield scrapy.Request(next_page, callback=self.parse)
```
Now, after extracting the data, the `parse()` method looks for the link to
the next page, builds a full absolute URL using the
`urljoin()` method (since the links can be
relative) and yields a new request to the next page, registering itself as
callback to handle the data extraction for the next page and to keep the
crawling going through all the pages.
What you see here is Scrapy’s mechanism of following links: when you yield
a Request in a callback method, Scrapy will schedule that request to be sent
and register a callback method to be executed when that request finishes.
Using this, you can build complex crawlers that follow links according to rules
you define, and extract different kinds of data depending on the page it’s
visiting.
In our example, it creates a sort of loop, following all the links to the next page
until it doesn’t find one – handy for crawling blogs, forums and other sites with
pagination.
#### A shortcut for creating Requests
As a shortcut for creating Request objects you can use
`response.follow`:
```python
import scrapy
class QuotesSpider(scrapy.Spider):
name = "quotes"
start_urls = [
"https://quotes.toscrape.com/page/1/",
]
def parse(self, response):
for quote in response.css("div.quote"):
yield {
"text": quote.css("span.text::text").get(),
"author": quote.css("span small::text").get(),
"tags": quote.css("div.tags a.tag::text").getall(),
}
next_page = response.css("li.next a::attr(href)").get()
if next_page is not None:
yield response.follow(next_page, callback=self.parse)
```
Unlike scrapy.Request, `response.follow` supports relative URLs directly - no
need to call urljoin. Note that `response.follow` just returns a Request
instance; you still have to yield this Request.
You can also pass a selector to `response.follow` instead of a string;
this selector should extract necessary attributes:
```python
for href in response.css("ul.pager a::attr(href)"):
yield response.follow(href, callback=self.parse)
```
For `` elements there is a shortcut: `response.follow` uses their href
attribute automatically. So the code can be shortened further:
```python
for a in response.css("ul.pager a"):
yield response.follow(a, callback=self.parse)
```
To create multiple requests from an iterable, you can use
`response.follow_all` instead:
```python
anchors = response.css("ul.pager a")
yield from response.follow_all(anchors, callback=self.parse)
```
or, shortening it further:
```python
yield from response.follow_all(css="ul.pager a", callback=self.parse)
```
#### More examples and patterns
Here is another spider that illustrates callbacks and following links,
this time for scraping author information:
```python
import scrapy
class AuthorSpider(scrapy.Spider):
name = "author"
start_urls = ["https://quotes.toscrape.com/"]
def parse(self, response):
author_page_links = response.css(".author + a")
yield from response.follow_all(author_page_links, self.parse_author)
pagination_links = response.css("li.next a")
yield from response.follow_all(pagination_links, self.parse)
def parse_author(self, response):
def extract_with_css(query):
return response.css(query).get(default="").strip()
yield {
"name": extract_with_css("h3.author-title::text"),
"birthdate": extract_with_css(".author-born-date::text"),
"bio": extract_with_css(".author-description::text"),
}
```
This spider will start from the main page, it will follow all the links to the
authors pages calling the `parse_author` callback for each of them, and also
the pagination links with the `parse` callback as we saw before.
Here we’re passing callbacks to
`response.follow_all` as positional
arguments to make the code shorter; it also works for
`Request`.
The `parse_author` callback defines a helper function to extract and cleanup the
data from a CSS query and yields the Python dict with the author data.
Another interesting thing this spider demonstrates is that, even if there are
many quotes from the same author, we don’t need to worry about visiting the
same author page multiple times. By default, Scrapy filters out duplicated
requests to URLs already visited, avoiding the problem of hitting servers too
much because of a programming mistake. This can be configured in the
`DUPEFILTER_CLASS` setting.
Hopefully by now you have a good understanding of how to use the mechanism
of following links and callbacks with Scrapy.
As yet another example spider that leverages the mechanism of following links,
check out the `CrawlSpider` class for a generic
spider that implements a small rules engine that you can use to write your
crawlers on top of it.
Also, a common pattern is to build an item with data from more than one page,
using a trick to pass additional data to the callbacks.
### Using spider arguments
You can provide command line arguments to your spiders by using the `-a`
option when running them:
```
scrapy crawl quotes -O quotes-humor.json -a tag=humor
```
These arguments are passed to the Spider’s `__init__` method and become
spider attributes by default.
In this example, the value provided for the `tag` argument will be available
via `self.tag`. You can use this to make your spider fetch only quotes
with a specific tag, building the URL based on the argument:
```python
import scrapy
class QuotesSpider(scrapy.Spider):
name = "quotes"
async def start(self):
url = "https://quotes.toscrape.com/"
tag = getattr(self, "tag", None)
if tag is not None:
url = url + "tag/" + tag
yield scrapy.Request(url, self.parse)
def parse(self, response):
for quote in response.css("div.quote"):
yield {
"text": quote.css("span.text::text").get(),
"author": quote.css("small.author::text").get(),
}
next_page = response.css("li.next a::attr(href)").get()
if next_page is not None:
yield response.follow(next_page, self.parse)
```
If you pass the `tag=humor` argument to this spider, you’ll notice that it
will only visit URLs from the `humor` tag, such as
`https://quotes.toscrape.com/tag/humor`.
You can learn more about handling spider arguments here.
### Next steps
This tutorial covered only the basics of Scrapy, but there’s a lot of other
features not mentioned here. Check the topics-whatelse section in the
intro-overview chapter for a quick overview of the most important ones.
You can continue from the section section-basics to know more about the
command-line tool, spiders, selectors and other things the tutorial hasn’t covered like
modeling the scraped data. If you’d prefer to play with an example project, check
the intro-examples section.
If you are using a coding agent, see agents.
Source: /intro/examples.md
## Examples
The best way to learn is with examples, and Scrapy is no exception. For this
reason, there is an example Scrapy project named [quotesbot](https://github.com/scrapy/quotesbot), that you can use to
play and learn more about Scrapy. It contains two spiders for
[https://quotes.toscrape.com](https://quotes.toscrape.com), one using CSS selectors and another one using XPath
expressions.
The [quotesbot](https://github.com/scrapy/quotesbot) project is available at: [https://github.com/scrapy/quotesbot](https://github.com/scrapy/quotesbot).
You can find more information about it in the project’s README.
If you’re familiar with git, you can checkout the code. Otherwise you can
download the project as a zip file by clicking
[here](https://github.com/scrapy/quotesbot/archive/master.zip).
Source: /topics/concepts.md
## Scrapy’s building blocks
Scrapy splits the work of crawling and scraping across a handful of pieces,
each responsible for one part of the problem. Placing a given piece of custom
logic where it belongs keeps that logic reusable across spiders and the
project easy to follow — this separation is what lets Scrapy scale from a
single spider to large, maintainable projects, rather than one script per
site.
This page is a quick map of those pieces, what problem each one solves, and
when to reach for it.
### Spiders
A spider defines which requests to send and how to
parse their responses. It is where a scraping project usually starts: one
spider per site (or group of similar sites), yielding items
and further requests as it parses each response.
### Items
An item is the data you extract, as a plain key-value
object. Spiders yield them, and everything downstream (item pipelines, feed
exports) works with them regardless of which item type a project uses.
### Settings
Settings configure everything else on this page:
which middlewares and extensions are enabled and in what order, how item
pipelines behave, feed export destinations, and Scrapy’s own defaults.
Settings are usually project-wide, but a spider can override them for itself
alone through `custom_settings`, which is often the
simplest way to turn something project-wide, such as a downloader middleware,
into behavior that only applies to the spiders that need it.
### Downloader middleware
A downloader middleware sees every
request Scrapy sends and every response it receives, regardless of which
spider is running — though a spider can narrow that to itself alone through
`custom_settings`. Reach for one when the logic is about
the HTTP layer itself, such as adding proxies or authentication headers,
retrying or redirecting based on the response, or short-circuiting a request
without hitting the network.
### Spider middleware
A spider middleware sits between the engine
and a spider’s callbacks, processing the responses going in and the items and
requests coming out. Reach for one to post-process what a spider produces
(filtering items, adding requests, handling exceptions) without changing the
spider’s own parsing code, especially when the same post-processing should
apply across several spiders.
### Item pipelines
An item pipeline processes items after a spider
yields them: cleaning up fields, validating data, dropping duplicates, or
storing items in a database. This is the usual place for anything that acts on
a *scraped item* rather than on a request or response.
### Feed exports
Feed exports write scraped items straight to a
file or storage (JSON, CSV, XML, and more) without any custom code. Reach for
them before writing an item pipeline if all you need is to dump the scraped
items somewhere.
### Extensions
An extension is the wildcard: use it for
functionality that does not fit any of the roles above, such as collecting
stats, enforcing a memory limit, or logging crawl progress. Extensions
typically hook into signals rather than into the
request/response/item flow.
### Signals
Signals notify your code when something happens during
a crawl, such as a spider opening or closing, or a response being downloaded.
Extensions are the usual place to connect to them, but any component can.
### Other components
The pieces above are what most projects touch directly. Scrapy has more of
them under the hood, such as download handlers, the scheduler, and feed
storages, that most projects never need to customize; see
topics-components for the full list.
Source: /topics/commands.md
## Command line tool
Scrapy is controlled through the `scrapy` command-line tool, to be referred to
here as the “Scrapy tool” to differentiate it from the sub-commands, which we
just call “commands” or “Scrapy commands”.
The Scrapy tool provides several commands, for multiple purposes, and each one
accepts a different set of arguments and options.
(The `scrapy deploy` command has been removed in 1.0 in favor of the
standalone `scrapyd-deploy`. See [Deploying your project](https://scrapyd.readthedocs.io/en/latest/deploy.html).)
### Configuration settings
Scrapy will look for configuration parameters in ini-style `scrapy.cfg` files
in standard locations:
1. `/etc/scrapy.cfg` or `c:\scrapy\scrapy.cfg` (system-wide),
2. `~/.config/scrapy.cfg` (`$XDG_CONFIG_HOME`) and `~/.scrapy.cfg` (`$HOME`)
for global (user-wide) settings, and
3. `scrapy.cfg` inside a Scrapy project’s root (see next section).
Settings from these files are merged in the listed order of preference:
user-defined values have higher priority than system-wide defaults
and project-wide settings will override all others, when defined.
Scrapy also understands, and can be configured through, a number of environment
variables. Currently these are:
* `SCRAPY_SETTINGS_MODULE` (see topics-settings-module-envvar)
* `SCRAPY_PROJECT` (see topics-project-envvar)
* `SCRAPY_PYTHON_SHELL` (see topics-shell)
### Default structure of Scrapy projects
Before delving into the command-line tool and its sub-commands, let’s first
understand the directory structure of a Scrapy project.
Though it can be modified, all Scrapy projects have the same file
structure by default, similar to this:
```none
scrapy.cfg
myproject/
__init__.py
items.py
middlewares.py
pipelines.py
settings.py
spiders/
__init__.py
spider1.py
spider2.py
...
```
The directory where the `scrapy.cfg` file resides is known as the *project
root directory*. That file contains the name of the python module that defines
the project settings. Here is an example:
```ini
[settings]
default = myproject.settings
```
The project name also appears elsewhere by default. `SPIDER_MODULES`
and `NEWSPIDER_MODULE` reference the project module itself, so they
must match its actual location. `BOT_NAME` defaults to the same name
but is just an identifier, and the capitalized project-name prefix of class
names in `middlewares.py` and `pipelines.py` is only a naming
convention; neither needs to match the module name.
#### Finding projects
To find the Scrapy projects in a directory tree, e.g. from an editor extension,
use `find_projects()`:
#### scrapy.utils.project.find_projects(path: str | os.PathLike[str] = '.', \*, max_depth: int | None = None, ignored_dirs: Iterable[str] = ()) → Iterator[Path]
Yield the root directory of every Scrapy project found in *path* or,
recursively, in any of its subdirectories.
> [!WARNING]
> Added in version 2.19.0.
Once a project is found, its subdirectories are not searched.
Hidden directories and any directory named in *ignored_dirs* are not
searched either, and neither are virtual environments, identified by a
`pyvenv.cfg` file, regardless of their directory name.
*max_depth* limits how deep to search: 0 only checks *path* itself, 1 also
checks its direct subdirectories, and so on. By default, the whole
directory tree is searched.
### Sharing the root directory between projects
A project root directory, the one that contains the `scrapy.cfg`, may be
shared by multiple Scrapy projects, each with its own settings module.
In that case, you must define one or more aliases for those settings modules
under `[settings]` in your `scrapy.cfg` file:
```ini
[settings]
default = myproject1.settings
project1 = myproject1.settings
project2 = myproject2.settings
```
By default, the `scrapy` command-line tool will use the `default` settings.
Use the `SCRAPY_PROJECT` environment variable to specify a different project
for `scrapy` to use:
```none
$ scrapy settings --get BOT_NAME
Project 1 Bot
$ export SCRAPY_PROJECT=project2
$ scrapy settings --get BOT_NAME
Project 2 Bot
```
### Using the `scrapy` tool
You can start by running the Scrapy tool with no arguments and it will print
some usage help and the available commands:
```none
Scrapy X.Y - no active project
Usage:
scrapy [options] [args]
Available commands:
fetch Fetch a URL using the Scrapy downloader
runspider Run a spider from a Python file, no project required
[...]
```
The first line will print the currently active project if you’re inside a
Scrapy project. In this example it was run from outside a project. If run from inside
a project it would have printed something like this:
```none
Scrapy X.Y - project: myproject
Usage:
scrapy [options] [args]
[...]
```
#### Creating projects
The first thing you typically do with the `scrapy` tool is create your Scrapy
project:
```none
scrapy startproject myproject [project_dir]
```
That will create a Scrapy project under the `project_dir` directory.
If `project_dir` wasn’t specified, `project_dir` will be the same as `myproject`.
Next, you go inside the new project directory:
```none
cd project_dir
```
And you’re ready to use the `scrapy` command to manage and control your
project from there.
#### Controlling projects
You use the `scrapy` tool from inside your projects to control and manage
them.
For example, to create a new spider:
```none
scrapy genspider mydomain mydomain.com
```
Some Scrapy commands (like `crawl`) must be run from inside a Scrapy
project. See the commands reference below for more
information on which commands must be run from inside projects, and which not.
Also keep in mind that some commands may have slightly different behaviours
when running them from inside projects. For example, the fetch command will use
spider-overridden behaviours (such as the `custom_settings` attribute to
override settings) if the url being fetched is associated with some specific
spider. This is intentional, as the `fetch` command is meant to be used to
check how spiders are downloading pages.
### Available tool commands
This section contains a list of the available built-in commands with a
description and some usage examples. Remember, you can always get more info
about each command by running:
```none
scrapy -h
```
And you can see all available commands with:
```none
scrapy -h
```
There are two kinds of commands, those that only work from inside a Scrapy
project (Project-specific commands) and those that also work without an active
Scrapy project (Global commands), though they may behave slightly differently
when run from inside a project (as they would use the project overridden
settings).
Global commands:
* `startproject`
* `genspider`
* `settings`
* `runspider`
* `shell`
* `fetch`
* `view`
* `version`
* `bench`
Project-only commands:
* `crawl`
* `check`
* `list`
* `edit`
* `parse`
#### startproject
* Syntax: `scrapy startproject [project_dir]`
* Requires project: *no*
Creates a new Scrapy project named `project_name`, under the `project_dir`
directory.
If `project_dir` wasn’t specified, `project_dir` will be the same as `project_name`.
Usage example:
```none
$ scrapy startproject myproject
```
#### genspider
* Syntax: `scrapy genspider [-t template] `
* Requires project: *no*
Creates a new spider in the current folder or in the current project’s `spiders` folder, if called from inside a project. The `` parameter is set as the spider’s `name`, while `` is used to generate the `allowed_domains` and `start_urls` spider’s attributes.
Usage example:
```none
$ scrapy genspider -l
Available templates:
basic
crawl
csvfeed
xmlfeed
$ scrapy genspider example example.com
Created spider 'example' using template 'basic'
$ scrapy genspider -t crawl scrapyorg scrapy.org
Created spider 'scrapyorg' using template 'crawl'
```
This is just a convenient shortcut command for creating spiders based on
pre-defined templates, but certainly not the only way to create spiders. You
can just create the spider source code files yourself, instead of using this
command.
##### Custom spider templates
To define your own spider templates, point `TEMPLATES_DIR` at a
directory with a `spiders` subdirectory, and write a `*name*.tmpl`
file there for every template, where *name* is the value to pass to `-t`.
Your templates replace the built-in ones, which live in the `templates`
directory of the `scrapy` package, so copy over any of those that you want to
keep.
Alternatively, pass `-t` a path to a `.tmpl` file instead of a name,
to use it without touching `TEMPLATES_DIR`.
Templates are rendered with `string.Template`: `$variable` and
`${variable}` are replaced, and `$$` renders as a single `$`, which
regular expressions often need. Rendering fails on any variable other than the
following:
- `name`: the spider name, as passed to the command.
- `module`: *name* as a valid module name, also used as the file name of
the generated spider.
- `classname`: *module* in camel case, with a `Spider` suffix.
- `url`: the URL passed to the command, with an `https` scheme added if
it had none.
- `domain`: the domain of *url*.
- `project_name`: `BOT_NAME`.
- `ProjectName`: *project_name* in camel case.
#### crawl
* Syntax: `scrapy crawl `
* Requires project: *yes*
Start crawling using the spider with the given `name`,
which must be one of those that `list` reports. To run a spider from a
file instead, use `runspider`.
Supported options:
* `-h, --help`: show a help message and exit
* `-a NAME=VALUE`: set a spider argument (may be repeated)
* `--output FILE` or `-o FILE`: append scraped items to the end of FILE (use - for stdout). To define the output format, set a colon at the end of the output URI (i.e. `-o FILE:FORMAT`)
* `--overwrite-output FILE` or `-O FILE`: dump scraped items into FILE, overwriting any existing file. To define the output format, set a colon at the end of the output URI (i.e. `-O FILE:FORMAT`)
Usage examples:
```none
$ scrapy crawl myspider
[ ... myspider starts crawling ... ]
$ scrapy crawl -o myfile:csv myspider
[ ... myspider starts crawling and appends the result to the file myfile in csv format ... ]
$ scrapy crawl -O myfile:json myspider
[ ... myspider starts crawling and saves the result in myfile in json format overwriting the original content... ]
```
#### check
* Syntax: `scrapy check [-l] `
* Requires project: *yes*
Run contract checks.
> [!WARNING]
> Added in version 2.19.0: The `-a` option, to pass spider arguments, as in `crawl`.
Usage examples:
```none
$ scrapy check -l
first_spider
* parse
* parse_item
second_spider
* parse
* parse_item
$ scrapy check
F.F.
======================================================================
FAIL: [first_spider] parse (@returns post-hook)
----------------------------------------------------------------------
Traceback (most recent call last):
...
scrapy.exceptions.ContractFail: Returned 92 requests, expected 0..4
======================================================================
FAIL: [first_spider] parse_item (@scrapes post-hook)
----------------------------------------------------------------------
Traceback (most recent call last):
...
scrapy.exceptions.ContractFail: Missing fields: RetailPricex
----------------------------------------------------------------------
Ran 4 contracts in 0.174s
FAILED (failures=2)
```
#### list
* Syntax: `scrapy list`
* Requires project: *yes*
List all available spiders in the current project. The output is one spider per
line.
Usage example:
```none
$ scrapy list
spider1
spider2
```
#### edit
* Syntax: `scrapy edit `
* Requires project: *yes*
Edit the given spider using the editor defined in the `EDITOR` environment
variable or (if unset) the `EDITOR` setting.
This command is provided only as a convenient shortcut for the most common
case, the developer is of course free to choose any tool or IDE to write and
debug spiders.
Usage example:
```none
$ scrapy edit spider1
```
#### fetch
* Syntax: `scrapy fetch `
* Requires project: *no*
Downloads the given URL using the Scrapy downloader and writes the contents to
standard output.
The interesting thing about this command is that it fetches the page the way the
spider would download it. For example, if the spider has a `USER_AGENT`
attribute which overrides the User Agent, it will use that one.
So this command can be used to “see” how your spider would fetch a certain page.
If used outside a project, no particular per-spider behaviour would be applied
and it will just use the default Scrapy downloader settings.
Supported options:
* `--spider=SPIDER`: bypass spider autodetection and force use of specific spider
* `--headers`: print the request’s and response’s HTTP headers instead of the response’s body
* `--no-redirect`: do not follow HTTP 3xx redirects (default is to follow them)
Usage examples:
```none
$ scrapy fetch --nolog http://www.example.com/some/page.html
[ ... html content here ... ]
$ scrapy fetch --nolog --headers http://www.example.com/
> Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
> Accept-Language: en
> User-Agent: Scrapy/2.16.0 (+https://scrapy.org)
> Accept-Encoding: gzip, deflate, br, zstd
>
< Date: Wed, 08 Jul 2026 06:15:01 GMT
< Content-Type: text/html
< Server: cloudflare
< Last-Modified: Wed, 01 Jul 2026 17:50:18 GMT
< Allow: GET, HEAD
< Cf-Cache-Status: HIT
< Age: 8184
< Cf-Ray: a17cf3b80eddf141-DME
```
#### view
* Syntax: `scrapy view `
* Requires project: *no*
Opens the given URL in a browser, as your Scrapy spider would “see” it.
Sometimes spiders see pages differently from regular users, so this can be used
to check what the spider “sees” and confirm it’s what you expect.
Supported options:
* `--spider=SPIDER`: bypass spider autodetection and force use of specific spider
* `--no-redirect`: do not follow HTTP 3xx redirects (default is to follow them)
Usage example:
```none
$ scrapy view http://www.example.com/some/page.html
[ ... browser starts ... ]
```
#### shell
* Syntax: `scrapy shell [url]`
* Requires project: *no*
Starts the Scrapy shell for the given URL (if given) or empty if no URL is
given. Also supports UNIX-style local file paths, either relative with
`./` or `../` prefixes or absolute file paths.
See topics-shell for more info.
Supported options:
* `--spider=SPIDER`: bypass spider autodetection and force use of specific spider
* `-c code`: evaluate the code in the shell, print the result and exit
* `--no-redirect`: do not follow HTTP 3xx redirects (default is to follow them);
this only affects the URL you may pass as argument on the command line;
once you are inside the shell, `fetch(url)` will still follow HTTP redirects by default.
Usage example:
```none
$ scrapy shell http://www.example.com/some/page.html
[ ... scrapy shell starts ... ]
$ scrapy shell --nolog http://www.example.com/ -c '(response.status, response.url)'
(200, 'http://www.example.com/')
# shell follows HTTP redirects by default
$ scrapy shell --nolog http://httpbin.org/redirect-to?url=http%3A%2F%2Fexample.com%2F -c '(response.status, response.url)'
(200, 'http://example.com/')
# you can disable this with --no-redirect
# (only for the URL passed as command line argument)
$ scrapy shell --no-redirect --nolog http://httpbin.org/redirect-to?url=http%3A%2F%2Fexample.com%2F -c '(response.status, response.url)'
(302, 'http://httpbin.org/redirect-to?url=http%3A%2F%2Fexample.com%2F')
```
#### parse
* Syntax: `scrapy parse [options]`
* Requires project: *yes*
Fetches the given URL and parses it with the spider that handles it, using the
method passed with the `--callback` option, or `parse` if not given.
Supported options:
* `--spider=SPIDER`: bypass spider autodetection and force use of specific spider
* `-a NAME=VALUE`: set spider argument (may be repeated)
* `--callback` or `-c`: spider method to use as callback for parsing the
response
* `--meta` or `-m`: additional request meta that will be passed to the callback
request. This must be a valid json string. Example: –meta=’{“foo” : “bar”}’
* `--cbkwargs`: additional keyword arguments that will be passed to the callback.
This must be a valid json string. Example: –cbkwargs=’{“foo” : “bar”}’
* `--pipelines`: process items through pipelines
* `--rules` or `-r`: use `CrawlSpider`
rules to discover the callback (i.e. spider method) to use for parsing the
response
* `--noitems`: don’t show scraped items
* `--nolinks`: don’t show extracted links
* `--nocolour`: avoid using pygments to colorize the output
* `--depth` or `-d`: depth level for which the requests should be followed
recursively (default: 1)
* `--verbose` or `-v`: display information for each depth level
* `--output` or `-o`: dump scraped items to a file
Usage example:
```none
$ scrapy parse http://www.example.com/ -c parse_item
[ ... scrapy log lines crawling example.com spider ... ]
>>> STATUS DEPTH LEVEL 1 <<<
# Scraped Items ------------------------------------------------------------
[{'name': 'Example item',
'category': 'Furniture',
'length': '12 cm'}]
# Requests -----------------------------------------------------------------
[]
```
#### settings
* Syntax: `scrapy settings [options]`
* Requires project: *no*
Get the value of a Scrapy setting.
If used inside a project it’ll show the project setting value, otherwise it’ll
show the default Scrapy value for that setting.
Example usage:
```none
$ scrapy settings --get BOT_NAME
scrapybot
$ scrapy settings --get DOWNLOAD_DELAY
0
```
#### runspider
* Syntax: `scrapy runspider `
* Requires project: *no*
Run the spider defined in the given Python file, without requiring a project.
Supported options: the same as `crawl`.
Example usage:
```none
$ scrapy runspider myspider.py
[ ... spider starts crawling ... ]
```
#### version
* Syntax: `scrapy version [-v]`
* Requires project: *no*
Prints the Scrapy version. If used with `-v` it also prints Python, Twisted
and Platform info, which is useful for bug reports.
#### bench
* Syntax: `scrapy bench`
* Requires project: *no*
Run a quick benchmark test. benchmarking.
### Commands that run a crawl
Many commands need to run a crawl of some kind, running either a user-provided
spider or a special internal one:
* `bench`
* `check`
* `crawl`
* `fetch`
* `parse`
* `runspider`
* `shell`
* `view`
They use an internal instance of `scrapy.crawler.AsyncCrawlerProcess` or
`scrapy.crawler.CrawlerProcess` for this. In most cases this detail
shouldn’t matter to the user running the command, but when the user needs
a non-default Twisted reactor, it may be important.
Scrapy decides which of these two classes to use based on the value of the
`TWISTED_REACTOR` and `TWISTED_REACTOR_ENABLED` settings.
With `TWISTED_REACTOR_ENABLED` set to `False` it will use
`AsyncCrawlerProcess`. Otherwise, if the
`TWISTED_REACTOR` value is the default one
(`'twisted.internet.asyncioreactor.AsyncioSelectorReactor'`),
`AsyncCrawlerProcess` will be used, otherwise
`CrawlerProcess` will be used. The spider settings are not taken into account when doing this, as they are
loaded after this decision is made. This may cause an error if the
project-level setting is set to the asyncio reactor
(explicitly or by using the Scrapy default) and the setting of the spider being run is set to a different one, because
`AsyncCrawlerProcess` only supports the asyncio reactor.
In this case you should set the `FORCE_CRAWLER_PROCESS` setting to
`True` (at the project level or via the command line) so that Scrapy uses
`CrawlerProcess` which supports all reactors.
### Custom project commands
You can also add your custom project commands by using the
`COMMANDS_MODULE` setting. This allows you to create project-specific
commands that are automatically discovered and made available through the
`scrapy` command-line tool.
#### Creating custom commands
To create a custom command, inherit from the `ScrapyCommand` class
and implement the required methods. This allows you to extend Scrapy’s command-line
interface with your own functionality, such as project-specific utilities, data
processing tools, or deployment helpers.
When you create a custom command, you define its behavior by setting class attributes
and overriding specific methods. Here’s what you need to know:
**Attributes you can set:**
* `requires_project` (bool): If `True`,
the command only runs inside a Scrapy project (default: `False`).
* `requires_crawler_process` (bool): If
`True`, a `AsyncCrawlerProcess` or
`CrawlerProcess` instance will be created by Scrapy
when the command runs and made available in the
`crawler_process` attribute (default:
`True`).
* `default_settings` (dict): Settings that
will override the default ones when running this command (default: `{}`).
* `exitcode` (int): Process exit code to
set when the command completes (default: `0`).
**Methods you must override:**
* `short_desc()`: Return a short description
of the command.
* `run()`: Main entry point for the command
execution.
**Methods you can override:**
* `syntax()`: Return command syntax
(preferably one-line, without command name).
* `long_desc()`: Return a detailed command
description.
* `add_options()`: Add command-specific
options to the argument parser.
* `process_options()`: Process parsed
command-line options and set settings before
`crawler_process` is instantiated.
**Example custom command:**
```python
from scrapy.commands import ScrapyCommand
import argparse
class MyCustomCommand(ScrapyCommand):
requires_project = True
def syntax(self):
return "[options] "
def short_desc(self):
return "Run my custom command"
def add_options(self, parser):
super().add_options(parser)
parser.add_argument("--my-option", help="My custom option")
def run(self, args, opts):
# Command implementation here
spider_name = args[0] if args else None
print(f"Running custom command for spider: {spider_name}")
```
For real examples, see the built-in Scrapy commands in the [scrapy/commands](https://github.com/scrapy/scrapy/tree/master/scrapy/commands) directory.
#### *class* scrapy.commands.ScrapyCommand
Base class for all Scrapy commands.
##### requires_project *: bool* *= False*
##### requires_crawler_process *: bool* *= True*
##### crawler_process *: CrawlerProcessBase | None* *= None*
##### default_settings *: ClassVar[dict[str, Any]]* *= {}*
##### exitcode *: int* *= 0*
##### set_crawler(crawler: Crawler) → None
##### syntax() → str
Command syntax (preferably one-line). Do not include command name.
##### *abstractmethod* short_desc() → str
A short description of the command.
##### long_desc() → str
A long description of the command. Return short description when not
available. It cannot contain newlines since contents will be formatted
by argparse which removes newlines and wraps text.
##### help() → str
##### add_options(parser: ArgumentParser) → None
Populate the option parser with the options available for this command.
##### process_options(args: list[str], opts: Namespace) → None
Set settings based on the command line options.
##### *abstractmethod* run(args: list[str], opts: Namespace) → None
Entry point for running commands.
> [!NOTE]
> This is a pre-crawler setting.
#### Register commands via setup.py entry points
You can also add Scrapy commands from an external library by adding a
`scrapy.commands` section in the entry points of the library `setup.py`
file.
The following example adds `my_command` command:
```python
from setuptools import setup, find_packages
setup(
name="scrapy-mymodule",
entry_points={
"scrapy.commands": [
"my_command=my_scrapy_module.commands:MyCommand",
],
},
)
```
#### COMMANDS_MODULE
Default: `''` (empty string)
A module to use for looking up custom Scrapy commands. This is used to add custom
commands for your Scrapy project.
Example:
```python
COMMANDS_MODULE = "mybot.commands"
```
Source: /topics/spiders.md
## Spiders
Spiders are classes that define how a site, or a group of sites, is scraped:
which requests to send, and how to parse their responses to extract data and to
send additional requests.
A crawl goes as follows:
1. Scrapy iterates the `start()` method of the spider to
get the initial requests. By default, that method yields a
`Request` object for each URL in
`start_urls`, with `parse()` as
callback.
2. Scrapy downloads each request and calls its callback with the resulting
`Response`.
3. Callbacks parse the response, typically using topics-selectors, and
return or yield item objects with the extracted data
and `Request` objects to continue the crawl, which go back
to step 2. See callback-output.
4. Items go through item pipelines, and are
usually stored through topics-feed-exports.
Scrapy includes different spider classes for different purposes, described
below.
### scrapy.Spider
##### *class* scrapy.spiders.Spider
#### *class* scrapy.Spider(\*args: Any, \*\*kwargs: Any)
Base class that any spider must subclass.
It provides a default `start()` implementation that sends
requests based on the `start_urls` class attribute and calls the
`parse()` method for each response.
##### name
A string which defines the name for this spider. The spider name is how
the spider is located (and instantiated) by Scrapy, so it must be
unique. However, nothing prevents you from instantiating more than one
instance of the same spider. This is the most important spider attribute
and it’s required.
If the spider scrapes a single domain, a common practice is to name the
spider after the domain, with or without the [TLD](https://en.wikipedia.org/wiki/Top-level_domain). So, for example, a
spider that crawls `mywebsite.com` would often be called
`mywebsite`.
##### allowed_domains
An optional list of strings containing domains that this spider is
allowed to crawl. Requests for URLs not belonging to the domain names
specified in this list (or their subdomains) won’t be followed if
`OffsiteMiddleware` is
enabled.
> [!WARNING]
> Changed in version 2.18.0: Changes to this attribute during a crawl are now taken into account.
Let’s say your target url is `https://www.example.com/1.html`,
then add `'example.com'` to the list.
You may modify this attribute while the spider runs, e.g. to allow
domains that you only learn about from an earlier response. The change
affects requests scheduled after it.
##### start_urls *: list[str]*
Start URLs. See `start()`.
##### custom_settings
A dictionary of settings that will be overridden from the project wide
configuration when running this spider. It must be defined as a class
attribute since the settings are updated before instantiation.
For a list of available built-in settings see:
topics-settings-ref.
##### crawler
This attribute is set by the `from_crawler()` class method after
initializing the class, and links to the
`Crawler` object to which this spider instance is
bound.
Crawlers encapsulate a lot of components in the project for their single
entry access (such as extensions, middlewares, signals managers, etc).
See topics-api-crawler to know more about them.
##### settings
Configuration for running this spider. This is a
`Settings` instance, see the
topics-settings topic for a detailed introduction on this subject.
##### logger
Python logger created with the Spider’s `name`. You can use it to
send log messages through it as described on
topics-logging-from-spiders.
##### state
A dict you can use to persist some spider state between batches.
See topics-keeping-persistent-state-between-batches to know more about it.
##### from_crawler(crawler, \*args, \*\*kwargs)
This is the class method used by Scrapy to create your spiders.
You probably won’t need to override this directly because the default
implementation acts as a proxy to the `__init__()` method, calling
it with the given arguments `args` and named arguments `kwargs`.
Nonetheless, this method sets the `crawler` and `settings`
attributes in the new instance so they can be accessed later inside the
spider’s code.
> [!WARNING]
> Changed in version 2.11: The settings in `crawler.settings` can now be modified in this
> method, which is handy if you want to modify them based on
> arguments. As a consequence, these settings aren’t the final values
> as they can be modified later by e.g. add-ons. For the same reason, most of the
> `Crawler` attributes aren’t initialized at
> this point.
>
> The final settings and the initialized
> `Crawler` attributes are available in the
> `start()` method, handlers of the
> `engine_started` signal and later.
* **Parameters:**
* **crawler** (`Crawler` instance) – crawler to which the spider will be bound
* **args** (*list*) – arguments passed to the `__init__()` method
* **kwargs** (*dict*) – keyword arguments passed to the `__init__()` method
##### *classmethod* update_settings(settings)
The `update_settings()` method is used to modify the spider’s settings
and is called during initialization of a spider instance.
It takes a `Settings` object as a parameter and
can add or update the spider’s configuration values. This method is a
class method, meaning that it is called on the `Spider`
class and allows all instances of the spider to share the same
configuration.
While per-spider settings can be set in
`custom_settings`, using `update_settings()`
allows you to dynamically add, remove or change settings based on other
settings, spider attributes or other factors and use setting priorities
other than `'spider'`. Also, it’s easy to extend `update_settings()`
in a subclass by overriding it, while doing the same with
`custom_settings` can be hard.
For example, suppose a spider needs to modify `FEEDS`:
```python
import scrapy
class MySpider(scrapy.Spider):
name = "myspider"
custom_feed = {
"/home/user/documents/items.json": {
"format": "json",
"indent": 4,
}
}
@classmethod
def update_settings(cls, settings):
super().update_settings(settings)
settings.setdefault("FEEDS", {}).update(cls.custom_feed)
```
##### *async* start() → AsyncIterator[Any]
Yield the initial `Request` objects to send.
> [!WARNING]
> Added in version 2.13.
For example:
```python
from scrapy import Request, Spider
class MySpider(Spider):
name = "myspider"
async def start(self):
yield Request("https://toscrape.com/")
```
The default implementation reads URLs from `start_urls` and
yields a request for each with `dont_filter`
enabled. It is functionally equivalent to:
```python
async def start(self):
for url in self.start_urls:
yield Request(url, dont_filter=True)
```
You can also yield items. For example:
```python
async def start(self):
yield {"foo": "bar"}
```
To write spiders that work on Scrapy versions lower than 2.13,
define also a synchronous `start_requests()` method that returns an
iterable. For example:
```python
def start_requests(self):
yield Request("https://toscrape.com/")
```
> [!NOTE]
> ### See also
>
> start-requests
##### parse(response: Response, \*\*kwargs: Any) → Any
Process *response*, i.e. extract data from it and generate new
requests.
This is the default callback: Scrapy uses
it for the response to any request that does not define a
`callback`, such as the requests that
`start()` yields by default.
Any `cb_kwargs` of the request are passed as
keyword parameters.
Spiders must define this method, unless every request that they
send defines a callback.
See callback-output about the supported return values.
##### closed(reason)
Called when the spider closes. This method provides a shortcut to
signals.connect() for the `spider_closed` signal.
Let’s see an example:
```python
import scrapy
class MySpider(scrapy.Spider):
name = "example.com"
allowed_domains = ["example.com"]
start_urls = [
"http://www.example.com/1.html",
"http://www.example.com/2.html",
"http://www.example.com/3.html",
]
def parse(self, response):
self.logger.info("A response from %s just arrived!", response.url)
```
Return multiple Requests and items from a single callback:
```python
import scrapy
class MySpider(scrapy.Spider):
name = "example.com"
allowed_domains = ["example.com"]
start_urls = [
"http://www.example.com/1.html",
"http://www.example.com/2.html",
"http://www.example.com/3.html",
]
def parse(self, response):
for h3 in response.xpath("//h3").getall():
yield {"title": h3}
for href in response.xpath("//a/@href").getall():
yield scrapy.Request(response.urljoin(href), self.parse)
```
Instead of `start_urls` you can use `start()`
directly; to give data more structure you can use `Item`
objects:
```python
import scrapy
from myproject.items import MyItem
class MySpider(scrapy.Spider):
name = "example.com"
allowed_domains = ["example.com"]
async def start(self):
yield scrapy.Request("http://www.example.com/1.html", self.parse)
yield scrapy.Request("http://www.example.com/2.html", self.parse)
yield scrapy.Request("http://www.example.com/3.html", self.parse)
def parse(self, response):
for h3 in response.xpath("//h3").getall():
yield MyItem(title=h3)
for href in response.xpath("//a/@href").getall():
yield scrapy.Request(response.urljoin(href), self.parse)
```
### Spider arguments
Spiders can receive arguments that modify their behaviour. Some common uses for
spider arguments are to define the start URLs or to restrict the crawl to
certain sections of the site, but they can be used to configure any
functionality of the spider.
Spider arguments are passed through the `crawl` command using the
`-a` option. For example:
```
scrapy crawl myspider -a category=electronics
```
Spiders can access arguments in their \_\_init_\_ methods:
```python
import scrapy
class MySpider(scrapy.Spider):
name = "myspider"
def __init__(self, category=None, *args, **kwargs):
super().__init__(*args, **kwargs)
self.start_urls = [f"http://www.example.com/categories/{category}"]
# ...
```
The default \_\_init_\_ method will take any spider arguments
and copy them to the spider as attributes.
The above example can also be written as follows:
```python
import scrapy
class MySpider(scrapy.Spider):
name = "myspider"
async def start(self):
yield scrapy.Request(f"http://www.example.com/categories/{self.category}")
```
If you are running Scrapy from a script, you can
specify spider arguments when calling
`CrawlerProcess.crawl` or
`CrawlerRunner.crawl`:
```python
process = CrawlerProcess()
process.crawl(MySpider, category="electronics")
```
Keep in mind that spider arguments are only strings.
The spider will not do any parsing on its own.
If you were to set the `start_urls` attribute from the command line,
you would have to parse it on your own into a list
using something like `ast.literal_eval()` or `json.loads()`
and then set it as an attribute.
Otherwise, you would cause iteration over a `start_urls` string
(a very common python pitfall)
resulting in each character being seen as a separate url.
Spider arguments can also be passed through the Scrapyd `schedule.json` API.
See [Scrapyd documentation](https://scrapyd.readthedocs.io/en/latest/).
#### scrapy-spider-metadata parameters
Another alternative to pass spider arguments is the library [scrapy-spider-metadata](https://scrapy-spider-metadata.readthedocs.io/en/latest/params.html).
This allows for Scrapy spiders to define, validate, document and pre-process
their arguments as Pydantic models.
The example shows how to define typed parameters where a string argument
is automatically converted to an integer:
```python
import scrapy
from pydantic import BaseModel
from scrapy_spider_metadata import Args
class MyParams(BaseModel):
pages: int
class BookSpider(Args[MyParams], scrapy.Spider):
name = "bookspider"
start_urls = ["http://books.toscrape.com/catalogue"]
async def start(self):
for start_url in self.start_urls:
for index in range(1, self.args.pages + 1):
yield scrapy.Request(f"{start_url}/page-{index}.html")
def parse(self, response):
book_links = response.css("article.product_pod h3 a::attr(href)").getall()
for book_link in book_links:
yield response.follow(book_link, self.parse_book)
def parse_book(self, response):
yield {
"title": response.css("h1::text").get(),
"price": response.css("p.price_color::text").get(),
}
```
This spider can be called from the command line:
```
scrapy crawl bookspider -a pages=2
```
### Start requests
**Start requests** are `Request` objects yielded from the
`start()` method of a spider or from the
`process_start()` method of a
spider middleware.
> [!NOTE]
> ### See also
>
> start-request-order
#### Delaying start request iteration
Scrapy iterates `start()` as fast as it yields, so all start
requests reach the scheduler early in the crawl, however many they are. To
minimize the number of requests in the scheduler at any given time, and with it
resource usage (memory, or disk when using `JOBDIR`), override
`start()` to pause its iteration whenever there are
scheduled requests:
```python
async def start(self):
async for item_or_request in super().start():
if self.crawler.engine.needs_backout():
await self.crawler.signals.wait_for(signals.scheduler_empty)
yield item_or_request
```
#### Handling start errors
An exception raised by `start()` ends its iteration, so any
remaining start items and requests are never sent. Scrapy logs the exception,
sends the `spider_error` signal, and, once the already scheduled
requests are done, closes the spider with the `start_error`
`finish_reason`.
> [!WARNING]
> Changed in version 2.18.0: The close reason used to be `finished`, and neither the
> `spider_error` signal nor the `spider_exceptions/count` stat
> reported the exception.
To keep the iteration going, catch the exception yourself:
```python
async def start(self):
for url in self.start_urls:
try:
request = Request(url)
except ValueError:
self.logger.exception(f"Skipping start URL {url}")
else:
yield request
```
To stop the crawl instead, and choose your own `finish_reason`, raise
`CloseSpider`.
### Generic Spiders
Scrapy comes with some useful generic spiders that you can use to subclass
your spiders from. Their aim is to provide convenient functionality for a few
common scraping cases, like following all links on a site based on certain
rules, crawling from [Sitemaps](https://www.sitemaps.org/index.html), or parsing an XML/CSV feed.
For the examples used in the following spiders, we’ll assume you have a project
with a `TestItem` declared in a `myproject.items` module:
```python
from dataclasses import dataclass
@dataclass
class TestItem:
id: str | None = None
name: str | None = None
description: str | None = None
```
#### CrawlSpider
#### *class* scrapy.spiders.CrawlSpider
This is the most commonly used spider for crawling regular websites, as it
provides a convenient mechanism for following links by defining a set of rules.
It may not be the best suited for your particular web sites or project, but
it’s generic enough for several cases, so you can start from it and override it
as needed for more custom functionality, or just implement your own spider.
Apart from the attributes inherited from Spider (that you must
specify), this class supports a new attribute:
##### rules
Which is a list of one (or more) `Rule` objects. Each `Rule`
defines a certain behaviour for crawling the site. Rules objects are
described below. If multiple rules match the same link, the first one
will be used, according to the order they’re defined in this attribute.
Requests generated from `rules` carry the index of the matching rule
within `rules` in their `rule`
`Request.meta` key. `CrawlSpider` needs
that key to dispatch the response to the right rule, so copying it into a
request generated by a different rule sends the response to the wrong
callback.
This spider also exposes an overridable method:
##### parse_start_url(response, \*\*kwargs)
This method is called for each response produced for the URLs in
the spider’s `start_urls` attribute. It allows to parse
the initial responses and must return either an
item object, a `Request`
object, or an iterable containing any of them.
##### Crawling rules
#### *class* scrapy.spiders.Rule(link_extractor: LinkExtractor | None = None, callback: CallbackT | str | None = None, cb_kwargs: dict[str, Any] | None = None, follow: bool | None = None, process_links: ProcessLinksT | str | None = None, process_request: ProcessRequestT | str | None = None, errback: Callable[[Failure], Any] | str | None = None)
`link_extractor` is a Link Extractor object which
defines how links will be extracted from each crawled page. Each produced link will
be used to generate a `Request` object, which will contain the
link’s text in its `meta` dictionary (under the `link_text` key).
If omitted, a default link extractor created with no arguments will be used,
resulting in all links being extracted.
`callback` is a callable or a string (in which case a method from the spider
object with that name will be used) to be called for each link extracted with
the specified link extractor. This callback receives a `Response`
as its first argument and must return either a single instance or an iterable of
item objects and/or `Request` objects
(or any subclass of them). As mentioned above, the received `Response`
object will contain the text of the link that produced the `Request`
in its `meta` dictionary (under the `link_text` key)
`cb_kwargs` is a dict containing the keyword arguments to be passed to the
callback function.
`follow` is a boolean which specifies if links should be followed from each
response extracted with this rule. If `callback` is None `follow` defaults
to `True`, otherwise it defaults to `False`.
`process_links` is a callable, or a string (in which case a method from the
spider object with that name will be used) which will be called for each list
of links extracted from each response using the specified `link_extractor`.
This is mainly used for filtering purposes.
`process_request` is a callable (or a string, in which case a method from
the spider object with that name will be used) which will be called for every
`Request` extracted by this rule. This callable should
take said request as first argument and the `Response`
from which the request originated as second argument. It must return a
`Request` object or `None` (to filter out the request).
Use `process_request` to set the `priority` of
requests generated by a rule, e.g. `process_request=lambda request,
response: request.replace(priority=10)`.
`errback` is a callable or a string (in which case a method from the spider
object with that name will be used) to be called if any exception is
raised while processing a request generated by the rule.
It receives a `Twisted Failure`
instance as first parameter.
> [!WARNING]
> Because of its internal implementation, you must explicitly set
> callbacks for new requests when writing `CrawlSpider`-based spiders;
> unexpected behaviour can occur otherwise.
##### CrawlSpider example
Let’s now take a look at an example CrawlSpider with rules:
```python
from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor
class MySpider(CrawlSpider):
name = "example.com"
allowed_domains = ["example.com"]
start_urls = ["http://www.example.com"]
rules = (
# Extract links matching 'category.php' (but not matching 'subsection.php')
# and follow links from them (since no callback means follow=True by default).
Rule(LinkExtractor(allow=(r"category\.php",), deny=(r"subsection\.php",))),
# Extract links matching 'item.php' and parse them with the spider's method parse_item
Rule(LinkExtractor(allow=(r"item\.php",)), callback="parse_item"),
)
def parse_item(self, response):
self.logger.info("Hi, this is an item page! %s", response.url)
item = {}
item["id"] = response.xpath('//td[@id="item_id"]/text()').re(r"ID: (\d+)")
item["name"] = response.xpath('//td[@id="item_name"]/text()').get()
item["description"] = response.xpath(
'//td[@id="item_description"]/text()'
).get()
item["link_text"] = response.meta["link_text"]
url = response.xpath('//td[@id="additional_data"]/@href').get()
return response.follow(
url, self.parse_additional_page, cb_kwargs=dict(item=item)
)
def parse_additional_page(self, response, item):
item["additional_data"] = response.xpath(
'//p[@id="additional_data"]/text()'
).get()
return item
```
This spider would start crawling example.com’s home page, collecting category
links, and item links, parsing the latter with the `parse_item` method. For
each item response, some data will be extracted from the HTML using XPath, and
a dictionary will be filled with it.
#### XMLFeedSpider
#### *class* scrapy.spiders.XMLFeedSpider
XMLFeedSpider is designed for parsing XML feeds by iterating through them by a
certain node name. The iterator can be chosen from: `iternodes`, `xml`,
and `html`. It’s recommended to use the `iternodes` iterator for
performance reasons, since the `xml` and `html` iterators generate the
whole DOM at once in order to parse it. However, using `html` as the
iterator may be useful when parsing XML with bad markup.
To set the iterator and the tag name, you must define the following class
attributes:
##### iterator
A string which defines the iterator to use. It can be either:
> - `'iternodes'` - a fast iterator based on `lxml`
> - `'html'` - an iterator which uses `Selector`.
> Keep in mind this uses DOM parsing and must load all DOM in memory
> which could be a problem for big feeds. It also parses the feed
> with an HTML parser, which can silently mangle tags that HTML
> treats as void elements, such as `
http://example.com/
```
With `sitemap_alternate_links` set, this would retrieve both URLs. With
`sitemap_alternate_links` disabled, only `http://example.com/` would be
retrieved.
Default is `sitemap_alternate_links` disabled.
##### sitemap_filter(entries)
This is a filter function that could be overridden to select sitemap entries
based on their attributes.
For example:
```xml
http://example.com/
2005-01-01
```
We can define a `sitemap_filter` function to filter `entries` by date:
```python
from datetime import datetime
from scrapy.spiders import SitemapSpider
class FilteredSitemapSpider(SitemapSpider):
name = "filtered_sitemap_spider"
allowed_domains = ["example.com"]
sitemap_urls = ["http://example.com/sitemap.xml"]
def sitemap_filter(self, entries):
for entry in entries:
date_time = datetime.strptime(entry["lastmod"], "%Y-%m-%d")
if date_time.year >= 2005:
yield entry
```
This would retrieve only `entries` modified on 2005 and the following
years.
Entries are dict objects extracted from the sitemap document.
Usually, the key is the tag name and the value is the text inside it.
It’s important to notice that:
- as the loc attribute is required, entries without this tag are discarded
- alternate links are stored in a list with the key `alternate`
(see `sitemap_alternate_links`)
- namespaces are removed, so lxml tags named as `{namespace}tagname` become only `tagname`
If you omit this method, all entries found in sitemaps will be
processed, observing other attributes and their settings.
##### SitemapSpider examples
Simplest example: process all urls discovered through sitemaps using the
`parse` callback:
```python
from scrapy.spiders import SitemapSpider
class MySpider(SitemapSpider):
sitemap_urls = ["http://www.example.com/sitemap.xml"]
def parse(self, response):
pass # ... scrape item here ...
```
Process some urls with certain callback and other urls with a different
callback:
```python
from scrapy.spiders import SitemapSpider
class MySpider(SitemapSpider):
sitemap_urls = ["http://www.example.com/sitemap.xml"]
sitemap_rules = [
("/product/", "parse_product"),
("/category/", "parse_category"),
]
def parse_product(self, response):
pass # ... scrape product ...
def parse_category(self, response):
pass # ... scrape category ...
```
Follow sitemaps defined in the [robots.txt](https://www.robotstxt.org/) file and only follow sitemaps
whose url contains `/sitemap_shop`:
```python
from scrapy.spiders import SitemapSpider
class MySpider(SitemapSpider):
sitemap_urls = ["http://www.example.com/robots.txt"]
sitemap_rules = [
("/shop/", "parse_shop"),
]
sitemap_follow = ["/sitemap_shops"]
def parse_shop(self, response):
pass # ... scrape shop here ...
```
Combine SitemapSpider with other sources of urls:
```python
from scrapy import Request
from scrapy.spiders import SitemapSpider
class MySpider(SitemapSpider):
sitemap_urls = ["http://www.example.com/robots.txt"]
sitemap_rules = [
("/shop/", "parse_shop"),
]
other_urls = ["http://www.example.com/about"]
async def start(self):
async for item_or_request in super().start():
yield item_or_request
for url in self.other_urls:
yield Request(url, self.parse_other)
def parse_shop(self, response):
pass # ... scrape shop here ...
def parse_other(self, response):
pass # ... scrape other here ...
```
Source: /topics/selectors.md
## Selectors
When you’re scraping web pages, the most common task you need to perform is
to extract data from the HTML source. There are several libraries available to
achieve this, such as:
- [BeautifulSoup](https://www.crummy.com/software/BeautifulSoup/) is a very popular web scraping library among Python
programmers which constructs a Python object based on the structure of the
HTML code and also deals with bad markup reasonably well, but it has one
drawback: it’s slow.
- [lxml](https://lxml.de/) is an XML parsing library (which also parses HTML) with a pythonic
API based on `ElementTree`. (lxml is not part of the Python
standard library.)
Scrapy comes with its own mechanism for extracting data. They’re called
selectors because they “select” certain parts of the HTML document specified
either by [XPath](https://www.w3.org/TR/xpath/all/) or [CSS](https://www.w3.org/TR/selectors) expressions.
[XPath](https://www.w3.org/TR/xpath/all/) is a language for selecting nodes in XML documents, which can also be
used with HTML. [CSS](https://www.w3.org/TR/selectors) is a language for applying styles to HTML documents. It
defines selectors to associate those styles with specific HTML elements.
> [!NOTE]
> Scrapy Selectors is a thin wrapper around [parsel](https://parsel.readthedocs.io/en/latest/) library; the purpose of
> this wrapper is to provide better integration with Scrapy Response objects.
>
> [parsel](https://parsel.readthedocs.io/en/latest/) is a stand-alone web scraping library which can be used without
> Scrapy. It uses [lxml](https://lxml.de/) library under the hood, and implements an
> easy API on top of lxml API. It means Scrapy selectors are very similar
> in speed and parsing accuracy to lxml.
### Using selectors
#### Constructing selectors
Response objects expose a `Selector` instance
on `.selector` attribute:
```pycon
>>> response.selector.xpath("//span/text()").get()
'good'
```
Querying responses using XPath and CSS is so common that responses include two
more shortcuts: `response.xpath()` and `response.css()`:
```pycon
>>> response.xpath("//span/text()").get()
'good'
>>> response.css("span::text").get()
'good'
```
Scrapy selectors are instances of `Selector` class
constructed by passing either `TextResponse` object or
markup as a string (in `text` argument).
Usually there is no need to construct Scrapy selectors manually:
`response` object is available in Spider callbacks, so in most cases
it is more convenient to use `response.css()` and `response.xpath()`
shortcuts. By using `response.selector` or one of these shortcuts
you can also ensure the response body is parsed only once.
But if required, it is possible to use `Selector` directly.
Constructing from text:
```pycon
>>> from scrapy.selector import Selector
>>> body = "good "
>>> Selector(text=body).xpath("//span/text()").get()
'good'
```
Constructing from response - `HtmlResponse` is one of
`TextResponse` subclasses:
```pycon
>>> from scrapy.selector import Selector
>>> from scrapy.http import HtmlResponse
>>> response = HtmlResponse(url="http://example.com", body=body, encoding="utf-8")
>>> Selector(response=response).xpath("//span/text()").get()
'good'
```
`Selector` automatically chooses the best parsing rules
(XML vs HTML) based on input type.
#### Using selectors
To explain how to use the selectors we’ll use the `Scrapy shell` (which
provides interactive testing) and an example page located in the Scrapy
documentation server:
> [https://docs.scrapy.org/en/latest/_static/selectors-sample1.md](https://docs.scrapy.org/en/latest/_static/selectors-sample1.md)
For the sake of completeness, here’s its full HTML code:
```html
Example website
```
First, let’s open the shell:
```sh
scrapy shell https://docs.scrapy.org/en/latest/_static/selectors-sample1.md
```
Then, after the shell loads, you’ll have the response available as `response`
shell variable, and its attached selector in `response.selector` attribute.
Since we’re dealing with HTML, the selector will automatically use an HTML parser.
So, by looking at the HTML code of that
page, let’s construct an XPath for selecting the text inside the title tag:
```pycon
>>> response.xpath("//title/text()")
[]
```
To actually extract the textual data, you must call the selector `.get()`
or `.getall()` methods, as follows:
```pycon
>>> response.xpath("//title/text()").getall()
['Example website']
>>> response.xpath("//title/text()").get()
'Example website'
```
`.get()` always returns a single result; if there are several matches,
content of a first match is returned; if there are no matches, None
is returned. `.getall()` returns a list with all results.
Notice that CSS selectors can select text or attribute nodes using CSS3
pseudo-elements:
```pycon
>>> response.css("title::text").get()
'Example website'
```
As you can see, `.xpath()` and `.css()` methods return a
`SelectorList` instance, which is a list of new
selectors. This API can be used for quickly selecting nested data:
```pycon
>>> response.css("img").xpath("@src").getall()
['image1_thumb.jpg',
'image2_thumb.jpg',
'image3_thumb.jpg',
'image4_thumb.jpg',
'image5_thumb.jpg']
```
If you want to extract only the first matched element, you can call the
selector `.get()` (or its alias `.extract_first()` commonly used in
previous Scrapy versions):
```pycon
>>> response.xpath('//div[@id="images"]/a/text()').get()
'Name: My image 1 '
```
It returns `None` if no element was found:
```pycon
>>> response.xpath('//div[@id="not-exists"]/text()').get() is None
True
```
A default return value can be provided as an argument, to be used instead
of `None`:
```pycon
>>> response.xpath('//div[@id="not-exists"]/text()').get(default="not-found")
'not-found'
```
Instead of using e.g. `'@src'` XPath it is possible to query for attributes
using `.attrib` property of a `Selector`:
```pycon
>>> [img.attrib["src"] for img in response.css("img")]
['image1_thumb.jpg',
'image2_thumb.jpg',
'image3_thumb.jpg',
'image4_thumb.jpg',
'image5_thumb.jpg']
```
As a shortcut, `.attrib` is also available on SelectorList directly;
it returns attributes for the first matching element:
```pycon
>>> response.css("img").attrib["src"]
'image1_thumb.jpg'
```
This is most useful when only a single result is expected, e.g. when selecting
by id, or selecting unique elements on a web page:
```pycon
>>> response.css("base").attrib["href"]
'http://example.com/'
```
Now we’re going to get the base URL and some image links:
```pycon
>>> response.xpath("//base/@href").get()
'http://example.com/'
>>> response.css("base::attr(href)").get()
'http://example.com/'
>>> response.css("base").attrib["href"]
'http://example.com/'
>>> response.xpath('//a[contains(@href, "image")]/@href').getall()
['image1.html',
'image2.html',
'image3.html',
'image4.html',
'image5.html']
>>> response.css("a[href*=image]::attr(href)").getall()
['image1.html',
'image2.html',
'image3.html',
'image4.html',
'image5.html']
>>> response.xpath('//a[contains(@href, "image")]/img/@src').getall()
['image1_thumb.jpg',
'image2_thumb.jpg',
'image3_thumb.jpg',
'image4_thumb.jpg',
'image5_thumb.jpg']
>>> response.css("a[href*=image] img::attr(src)").getall()
['image1_thumb.jpg',
'image2_thumb.jpg',
'image3_thumb.jpg',
'image4_thumb.jpg',
'image5_thumb.jpg']
```
#### Extensions to CSS Selectors
Per W3C standards, [CSS selectors](https://www.w3.org/TR/selectors-3/#selectors) do not support selecting text nodes
or attribute values.
But selecting these is so essential in a web scraping context
that Scrapy (parsel) implements a couple of **non-standard pseudo-elements**:
* to select text nodes, use `::text`
* to select attribute values, use `::attr(name)` where *name* is the
name of the attribute that you want the value of
> [!WARNING]
> These pseudo-elements are Scrapy-/Parsel-specific.
> They will most probably not work with other libraries like
> [lxml](https://lxml.de/) or [PyQuery](https://pypi.org/project/pyquery/).
Examples:
* `title::text` selects children text nodes of a descendant `` element:
```pycon
>>> response.css("title::text").get()
'Example website'
```
* `*::text` selects all descendant text nodes of the current selector context:
```pycon
>>> response.css("#images *::text").getall()
['\n ',
'Name: My image 1 ',
'\n ',
'Name: My image 2 ',
'\n ',
'Name: My image 3 ',
'\n ',
'Name: My image 4 ',
'\n ',
'Name: My image 5 ',
'\n ']
```
* `foo::text` returns no results if `foo` element exists, but contains
no text (i.e. text is empty):
```pycon
>>> response.css("img::text").getall()
[]
This means ``.css('foo::text').get()`` could return None even if an element
exists. Use ``default=''`` if you always want a string:
```
```pycon
>>> response.css("img::text").get()
>>> response.css("img::text").get(default="")
''
```
* `a::attr(href)` selects the *href* attribute value of descendant links:
```pycon
>>> response.css("a::attr(href)").getall()
['image1.html',
'image2.html',
'image3.html',
'image4.html',
'image5.html']
```
> [!NOTE]
> See also: selecting-attributes.
> [!NOTE]
> You cannot chain these pseudo-elements. But in practice it would not
> make much sense: text nodes do not have attributes, and attribute values
> are string values already and do not have children nodes.
#### Nesting selectors
The selection methods (`.xpath()` or `.css()`) return a list of selectors
of the same type, so you can call the selection methods for those selectors
too. Here’s an example:
```pycon
>>> links = response.xpath('//a[contains(@href, "image")]')
>>> links.getall()
['<a href="image1.html">Name: My image 1 <br><img src="image1_thumb.jpg" alt="image1"></a>',
'<a href="image2.html">Name: My image 2 <br><img src="image2_thumb.jpg" alt="image2"></a>',
'<a href="image3.html">Name: My image 3 <br><img src="image3_thumb.jpg" alt="image3"></a>',
'<a href="image4.html">Name: My image 4 <br><img src="image4_thumb.jpg" alt="image4"></a>',
'<a href="image5.html">Name: My image 5 <br><img src="image5_thumb.jpg" alt="image5"></a>']
>>> for index, link in enumerate(links):
... href_xpath = link.xpath("@href").get()
... img_xpath = link.xpath("img/@src").get()
... print(f"Link number {index} points to url {href_xpath!r} and image {img_xpath!r}")
...
Link number 0 points to url 'image1.html' and image 'image1_thumb.jpg'
Link number 1 points to url 'image2.html' and image 'image2_thumb.jpg'
Link number 2 points to url 'image3.html' and image 'image3_thumb.jpg'
Link number 3 points to url 'image4.html' and image 'image4_thumb.jpg'
Link number 4 points to url 'image5.html' and image 'image5_thumb.jpg'
```
#### Selecting element attributes
There are several ways to get a value of an attribute. First, one can use
XPath syntax:
```pycon
>>> response.xpath("//a/@href").getall()
['image1.html', 'image2.html', 'image3.html', 'image4.html', 'image5.html']
```
XPath syntax has a few advantages: it is a standard XPath feature, and
`@attributes` can be used in other parts of an XPath expression - e.g.
it is possible to filter by attribute value.
Scrapy also provides an extension to CSS selectors (`::attr(...)`)
which allows to get attribute values:
```pycon
>>> response.css("a::attr(href)").getall()
['image1.html', 'image2.html', 'image3.html', 'image4.html', 'image5.html']
```
In addition to that, there is a `.attrib` property of Selector.
You can use it if you prefer to lookup attributes in Python
code, without using XPaths or CSS extensions:
```pycon
>>> [a.attrib["href"] for a in response.css("a")]
['image1.html', 'image2.html', 'image3.html', 'image4.html', 'image5.html']
```
This property is also available on SelectorList; it returns a dictionary
with attributes of a first matching element. It is convenient to use when
a selector is expected to give a single result (e.g. when selecting by element
ID, or when selecting an unique element on a page):
```pycon
>>> response.css("base").attrib
{'href': 'http://example.com/'}
>>> response.css("base").attrib["href"]
'http://example.com/'
```
`.attrib` property of an empty SelectorList is empty:
```pycon
>>> response.css("foo").attrib
{}
```
#### Using selectors with regular expressions
`Selector` also has a `.re()` method for extracting
data using regular expressions. However, unlike using `.xpath()` or
`.css()` methods, `.re()` returns a list of strings. So you
can’t construct nested `.re()` calls.
Here’s an example used to extract image names from the HTML code above:
```pycon
>>> response.xpath('//a[contains(@href, "image")]/text()').re(r"Name:\s*(.*)")
['My image 1 ',
'My image 2 ',
'My image 3 ',
'My image 4 ',
'My image 5 ']
```
There’s an additional helper reciprocating `.get()` (and its
alias `.extract_first()`) for `.re()`, named `.re_first()`.
Use it to extract just the first matching string:
```pycon
>>> response.xpath('//a[contains(@href, "image")]/text()').re_first(r"Name:\s*(.*)")
'My image 1 '
```
#### extract() and extract_first()
If you’re a long-time Scrapy user, you’re probably familiar
with `.extract()` and `.extract_first()` selector methods. Many blog posts
and tutorials are using them as well. These methods are still supported
by Scrapy, there are **no plans** to deprecate them.
However, Scrapy usage docs are now written using `.get()` and
`.getall()` methods. We feel that these new methods result in a more concise
and readable code.
The following examples show how these methods map to each other.
1. `SelectorList.get()` is the same as `SelectorList.extract_first()`:
```pycon
>>> response.css("a::attr(href)").get()
'image1.html'
>>> response.css("a::attr(href)").extract_first()
'image1.html'
```
1. `SelectorList.getall()` is the same as `SelectorList.extract()`:
```pycon
>>> response.css("a::attr(href)").getall()
['image1.html', 'image2.html', 'image3.html', 'image4.html', 'image5.html']
>>> response.css("a::attr(href)").extract()
['image1.html', 'image2.html', 'image3.html', 'image4.html', 'image5.html']
```
1. `Selector.get()` is the same as `Selector.extract()`:
```pycon
>>> response.css("a::attr(href)")[0].get()
'image1.html'
>>> response.css("a::attr(href)")[0].extract()
'image1.html'
```
1. For consistency, there is also `Selector.getall()`, which returns a list:
```pycon
>>> response.css("a::attr(href)")[0].getall()
['image1.html']
```
So, the main difference is that output of `.get()` and `.getall()` methods
is more predictable: `.get()` always returns a single result, `.getall()`
always returns a list of all extracted results. With `.extract()` method
it was not always obvious if a result is a list or not; to get a single
result either `.extract()` or `.extract_first()` should be called.
### Working with XPaths
Here are some tips which may help you to use XPath with Scrapy selectors
effectively. If you are not much familiar with XPath yet,
you may want to take a look first at this [XPath tutorial](http://www.zvon.org/comp/r/tut-XPath_1.html).
> [!NOTE]
> Some of the tips are based on [this post from Zyte’s blog](https://www.zyte.com/blog/xpath-tips-from-the-web-scraping-trenches/).
#### Working with relative XPaths
Keep in mind that if you are nesting selectors and use an XPath that starts
with `/`, that XPath will be absolute to the document and not relative to the
`Selector` you’re calling it from.
For example, suppose you want to extract all `<p>` elements inside `<div>`
elements. First, you would get all `<div>` elements:
```pycon
>>> divs = response.xpath("//div")
```
At first, you may be tempted to use the following approach, which is wrong, as
it actually extracts all `<p>` elements from the document, not only those
inside `<div>` elements:
```pycon
>>> for p in divs.xpath("//p"): # this is wrong - gets all <p> from the whole document
... print(p.get())
...
```
This is the proper way to do it (note the dot prefixing the `.//p` XPath):
```pycon
>>> for p in divs.xpath(".//p"): # extracts all <p> inside
... print(p.get())
...
```
Another common case would be to extract all direct `<p>` children:
```pycon
>>> for p in divs.xpath("p"):
... print(p.get())
...
```
For more details about relative XPaths see the [Location Paths](https://www.w3.org/TR/xpath-10/#location-paths) section in the
XPath specification.
#### When querying by class, consider using CSS
Because an element can contain multiple CSS classes, the XPath way to select elements
by class is the rather verbose:
```
*[contains(concat(' ', normalize-space(@class), ' '), ' someclass ')]
```
If you use `@class='someclass'` you may end up missing elements that have
other classes, and if you just use `contains(@class, 'someclass')` to make up
for that you may end up with more elements that you want, if they have a different
class name that shares the string `someclass`.
As it turns out, Scrapy selectors allow you to chain selectors, so most of the time
you can just select by class using CSS and then switch to XPath when needed:
```pycon
>>> from scrapy import Selector
>>> sel = Selector(
... text='<div class="hero shout"><time datetime="2014-07-23 19:00">Special date</time></div>'
... )
>>> sel.css(".shout").xpath("./time/@datetime").getall()
['2014-07-23 19:00']
```
This is cleaner than using the verbose XPath trick shown above. Just remember
to use the `.` in the XPath expressions that will follow.
#### Beware of the difference between //node[1] and (//node)[1]
`//node[1]` selects all the nodes occurring first under their respective parents.
`(//node)[1]` selects all the nodes in the document, and then gets only the first of them.
Example:
```pycon
>>> from scrapy import Selector
>>> sel = Selector(text="""
... <ul class="list">
... <li>1</li>
... <li>2</li>
... <li>3</li>
... </ul>
... <ul class="list">
... <li>4</li>
... <li>5</li>
... <li>6</li>
... </ul>""")
...
>>> xp = lambda x: sel.xpath(x).getall()
```
This gets all first `<li>` elements under whatever it is its parent:
```pycon
>>> xp("//li[1]")
['<li>1</li>', '<li>4</li>']
```
And this gets the first `<li>` element in the whole document:
```pycon
>>> xp("(//li)[1]")
['<li>1</li>']
```
This gets all first `<li>` elements under an `<ul>` parent:
```pycon
>>> xp("//ul/li[1]")
['<li>1</li>', '<li>4</li>']
```
And this gets the first `<li>` element under an `<ul>` parent in the whole document:
```pycon
>>> xp("(//ul/li)[1]")
['<li>1</li>']
```
#### Using text nodes in a condition
When you need to use the text content as argument to an [XPath string function](https://www.w3.org/TR/xpath-10/#section-String-Functions),
avoid using `.//text()` and use just `.` instead.
This is because the expression `.//text()` yields a collection of text elements – a *node-set*.
And when a node-set is converted to a string, which happens when it is passed as argument to
a string function like `contains()` or `starts-with()`, it results in the text for the first element only.
Example:
```pycon
>>> from scrapy import Selector
>>> sel = Selector(
... text='<a href="#">Click here to go to the <strong>Next Page</strong></a>'
... )
```
Converting a *node-set* to string:
```pycon
>>> sel.xpath("//a//text()").getall() # take a peek at the node-set
['Click here to go to the ', 'Next Page']
>>> sel.xpath("string(//a[1]//text())").getall() # convert it to string
['Click here to go to the ']
```
A *node* converted to a string, however, puts together the text of itself plus of all its descendants:
```pycon
>>> sel.xpath("//a[1]").getall() # select the first node
['<a href="#">Click here to go to the <strong>Next Page</strong></a>']
>>> sel.xpath("string(//a[1])").getall() # convert it to string
['Click here to go to the Next Page']
```
So, using the `.//text()` node-set won’t select anything in this case:
```pycon
>>> sel.xpath("//a[contains(.//text(), 'Next Page')]").getall()
[]
```
But using the `.` to mean the node, works:
```pycon
>>> sel.xpath("//a[contains(., 'Next Page')]").getall()
['<a href="#">Click here to go to the <strong>Next Page</strong></a>']
```
#### Variables in XPath expressions
XPath allows you to reference variables in your XPath expressions, using
the `$somevariable` syntax. This is somewhat similar to parameterized
queries or prepared statements in the SQL world where you replace
some arguments in your queries with placeholders like `?`,
which are then substituted with values passed with the query.
Here’s an example to match an element based on its “id” attribute value,
without hard-coding it (that was shown previously):
```pycon
>>> # `$val` used in the expression, a `val` argument needs to be passed
>>> response.xpath("//div[@id=$val]/a/text()", val="images").get()
'Name: My image 1 '
```
Here’s another example, to find the “id” attribute of a `<div>` tag containing
five `<a>` children (here we pass the value `5` as an integer):
```pycon
>>> response.xpath("//div[count(a)=$cnt]/@id", cnt=5).get()
'images'
```
All variable references must have a binding value when calling `.xpath()`
(otherwise you’ll get a `ValueError: XPath error:` exception).
This is done by passing as many named arguments as necessary.
[parsel](https://parsel.readthedocs.io/en/latest/), the library powering Scrapy selectors, has more details and examples
on [XPath variables](https://parsel.readthedocs.io/en/latest/usage.html#variables-in-xpath-expressions).
#### Removing namespaces
When dealing with scraping projects, it is often quite convenient to get rid of
namespaces altogether and just work with element names, to write more
simple/convenient XPaths. You can use the
`Selector.remove_namespaces()` method for that.
Let’s show an example that illustrates this with the Python Insider blog atom feed.
First, we open the shell with the url we want to scrape:
```sh
$ scrapy shell https://feeds.feedburner.com/PythonInsider
```
This is how the file starts:
```sh
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet ...
<feed xmlns="http://www.w3.org/2005/Atom"
xmlns:openSearch="http://a9.com/-/spec/opensearchrss/1.0/"
xmlns:blogger="http://schemas.google.com/blogger/2008"
xmlns:georss="http://www.georss.org/georss"
xmlns:gd="http://schemas.google.com/g/2005"
xmlns:thr="http://purl.org/syndication/thread/1.0"
xmlns:feedburner="http://rssnamespace.org/feedburner/ext/1.0">
...
```
You can see several namespace declarations including a default
`"http://www.w3.org/2005/Atom"` and another one using the `gd:` prefix for
`"http://schemas.google.com/g/2005"`.
Once in the shell we can try selecting all `<link>` objects and see that it
doesn’t work (because the Atom XML namespace is obfuscating those nodes):
```pycon
>>> response.xpath("//link")
[]
```
But once we call the `Selector.remove_namespaces()` method, all
nodes can be accessed directly by their names:
```pycon
>>> response.selector.remove_namespaces()
>>> response.xpath("//link")
[<Selector query='//link' data='<link rel="alternate" type="text/html" h'>,
<Selector query='//link' data='<link rel="next" type="application/atom+'>,
...
```
If you wonder why the namespace removal procedure isn’t always called by default
instead of having to call it manually, this is because of two reasons, which, in order
of relevance, are:
1. Removing namespaces requires to iterate and modify all nodes in the
document, which is a reasonably expensive operation to perform by default
for all documents crawled by Scrapy
2. There could be some cases where using namespaces is actually required, in
case some element names clash between namespaces. These cases are very rare
though.
#### Using EXSLT extensions
Being built atop [lxml](https://lxml.de/), Scrapy selectors support some [EXSLT](https://exslt.github.io/) extensions
and come with these pre-registered namespaces to use in XPath expressions:
| prefix | namespace | usage |
|----------|--------------------------------------|------------------------------------------------------------------|
| re | http://exslt.org/regular-expressions | [regular expressions](https://exslt.github.io/regexp/index.html) |
| set | http://exslt.org/sets | [set manipulation](https://exslt.github.io/set/index.html) |
##### Regular expressions
The `test()` function, for example, can prove quite useful when XPath’s
`starts-with()` or `contains()` are not sufficient.
Example selecting links in list item with a “class” attribute ending with a digit:
```pycon
>>> from scrapy import Selector
>>> doc = """
... <div>
... <ul>
... <li class="item-0"><a href="link1.html">first item</a></li>
... <li class="item-1"><a href="link2.html">second item</a></li>
... <li class="item-inactive"><a href="link3.html">third item</a></li>
... <li class="item-1"><a href="link4.html">fourth item</a></li>
... <li class="item-0"><a href="link5.html">fifth item</a></li>
... </ul>
... </div>
... """
>>> sel = Selector(text=doc, type="html")
>>> sel.xpath("//li//@href").getall()
['link1.html', 'link2.html', 'link3.html', 'link4.html', 'link5.html']
>>> sel.xpath(r'//li[re:test(@class, "item-\d$")]//@href').getall()
['link1.html', 'link2.html', 'link4.html', 'link5.html']
```
> [!WARNING]
> C library `libxslt` doesn’t natively support EXSLT regular
> expressions so [lxml](https://lxml.de/)’s implementation uses hooks to Python’s `re` module.
> Thus, using regexp functions in your XPath expressions may add a small
> performance penalty.
##### Set operations
These can be handy for excluding parts of a document tree before
extracting text elements for example.
Example extracting microdata (sample content taken from [https://schema.org/Product](https://schema.org/Product))
with groups of itemscopes and corresponding itemprops:
```pycon
>>> doc = """
... <div itemscope itemtype="http://schema.org/Product">
... <span itemprop="name">Kenmore White 17" Microwave</span>
... <img src="kenmore-microwave-17in.jpg" alt='Kenmore 17" Microwave' />
... <div itemprop="aggregateRating"
... itemscope itemtype="http://schema.org/AggregateRating">
... Rated <span itemprop="ratingValue">3.5</span>/5
... based on <span itemprop="reviewCount">11</span> customer reviews
... </div>
... <div itemprop="offers" itemscope itemtype="http://schema.org/Offer">
... <span itemprop="price">$55.00</span>
... <link itemprop="availability" href="http://schema.org/InStock" />In stock
... </div>
... Product description:
... <span itemprop="description">0.7 cubic feet countertop microwave.
... Has six preset cooking categories and convenience features like
... Add-A-Minute and Child Lock.</span>
... Customer reviews:
... <div itemprop="review" itemscope itemtype="http://schema.org/Review">
... <span itemprop="name">Not a happy camper</span> -
... by <span itemprop="author">Ellie</span>,
... <meta itemprop="datePublished" content="2011-04-01">April 1, 2011
... <div itemprop="reviewRating" itemscope itemtype="http://schema.org/Rating">
... <meta itemprop="worstRating" content = "1">
... <span itemprop="ratingValue">1</span>/
... <span itemprop="bestRating">5</span>stars
... </div>
... <span itemprop="description">The lamp burned out and now I have to replace
... it. </span>
... </div>
... <div itemprop="review" itemscope itemtype="http://schema.org/Review">
... <span itemprop="name">Value purchase</span> -
... by <span itemprop="author">Lucas</span>,
... <meta itemprop="datePublished" content="2011-03-25">March 25, 2011
... <div itemprop="reviewRating" itemscope itemtype="http://schema.org/Rating">
... <meta itemprop="worstRating" content = "1"/>
... <span itemprop="ratingValue">4</span>/
... <span itemprop="bestRating">5</span>stars
... </div>
... <span itemprop="description">Great microwave for the price. It is small and
... fits in my apartment.</span>
... </div>
... ...
... </div>
... """
>>> sel = Selector(text=doc, type="html")
>>> for scope in sel.xpath("//div[@itemscope]"):
... print("current scope:", scope.xpath("@itemtype").getall())
... props = scope.xpath("""
... set:difference(./descendant::*/@itemprop,
... .//*[@itemscope]/*/@itemprop)""")
... print(f" properties: {props.getall()}")
... print("")
...
current scope: ['http://schema.org/Product']
properties: ['name', 'aggregateRating', 'offers', 'description', 'review', 'review']
current scope: ['http://schema.org/AggregateRating']
properties: ['ratingValue', 'reviewCount']
current scope: ['http://schema.org/Offer']
properties: ['price', 'availability']
current scope: ['http://schema.org/Review']
properties: ['name', 'author', 'datePublished', 'reviewRating', 'description']
current scope: ['http://schema.org/Rating']
properties: ['worstRating', 'ratingValue', 'bestRating']
current scope: ['http://schema.org/Review']
properties: ['name', 'author', 'datePublished', 'reviewRating', 'description']
current scope: ['http://schema.org/Rating']
properties: ['worstRating', 'ratingValue', 'bestRating']
```
Here we first iterate over `itemscope` elements, and for each one,
we look for all `itemprops` elements and exclude those that are themselves
inside another `itemscope`.
#### Other XPath extensions
Scrapy selectors also provide a sorely missed XPath extension function
`has-class` that returns `True` for nodes that have all of the specified
HTML classes.
For the following HTML:
```pycon
>>> from scrapy.http import HtmlResponse
>>> response = HtmlResponse(
... url="http://example.com",
... body="""
... <html>
... <body>
... <p class="foo bar-baz">First</p>
... <p class="foo">Second</p>
... <p class="bar">Third</p>
... <p>Fourth</p>
... </body>
... </html>
... """,
... encoding="utf-8",
... )
```
You can use it like this:
```pycon
>>> response.xpath('//p[has-class("foo")]')
[<Selector query='//p[has-class("foo")]' data='<p class="foo bar-baz">First</p>'>,
<Selector query='//p[has-class("foo")]' data='<p class="foo">Second</p>'>]
>>> response.xpath('//p[has-class("foo", "bar-baz")]')
[<Selector query='//p[has-class("foo", "bar-baz")]' data='<p class="foo bar-baz">First</p>'>]
>>> response.xpath('//p[has-class("foo", "bar")]')
[]
```
So XPath `//p[has-class("foo", "bar-baz")]` is roughly equivalent to CSS
`p.foo.bar-baz`. Please note, that it is slower in most of the cases,
because it’s a pure-Python function that’s invoked for every node in question
whereas the CSS lookup is translated into XPath and thus runs more efficiently,
so performance-wise its uses are limited to situations that are not easily
described with CSS selectors.
Parsel also simplifies adding your own XPath extensions with
`set_xpathfunc()`.
### Built-in Selectors reference
#### Selector objects
#### *class* scrapy.Selector(\*args: Any, \*\*kwargs: Any)
An instance of `Selector` is a wrapper over response to select
certain parts of its content.
> [!WARNING]
> Changed in version 2.18.0: The type of a `JsonResponse` selector is now
> `json`, and the type of the selector of any other response that is
> neither HTML nor XML is determined from the response body.
`response` is an `HtmlResponse` or an
`XmlResponse` object that will be used for selecting
and extracting data.
`text` is a unicode string or utf-8 encoded text for cases when a
`response` isn’t available. Using `text` and `response` together is
undefined behavior.
`type` defines the selector type, it can be `"html"`, `"xml"`,
`"json"`, `"text"` or `None` (default). It’s passed to
`parsel.Selector` and its meaning is defined there. However, when
`type` is `None`, it is set to `"xml"` for an
`XmlResponse`, to `"json"` for a
`JsonResponse` and to `"html"` for an
`HtmlResponse` or for `text` before passing it to
`parsel.Selector`, which for any other response is left to
determine the type from the response body.
The response class, and hence the selector type, comes from the content
type that the website reports. When a website reports the wrong content
type, recast the response into the right class:
```python
response = response.replace(cls=HtmlResponse)
```
> [!NOTE]
> JSON selector support requires `parsel` 1.8.0 or higher. With
> older versions setting `type` to `"json"` or `"text"` is not
> supported.
##### xpath(query: str, namespaces: Mapping[str, str] | None = None, \*\*kwargs: Any) → SelectorList[Self]
Find nodes matching the xpath `query` and return the result as a
`SelectorList` instance with all elements flattened. List
elements implement `Selector` interface too.
`query` is a string containing the XPATH query to apply.
`namespaces` is an optional `prefix: namespace-uri` mapping (dict)
for additional prefixes to those registered with `register_namespace(prefix, uri)`.
Contrary to `register_namespace()`, these prefixes are not
saved for future calls.
Any additional named arguments can be used to pass values for XPath
variables in the XPath expression, e.g.:
```
selector.xpath('//a[href=$url]', url="http://www.example.com")
```
> [!NOTE]
> For convenience, this method can be called as `response.xpath()`
##### css(query: str) → SelectorList[Self]
Apply the given CSS selector and return a `SelectorList` instance.
`query` is a string containing the CSS selector to apply.
In the background, CSS queries are translated into XPath queries using
[cssselect](https://pypi.python.org/pypi/cssselect/) library and run `.xpath()` method.
> [!NOTE]
> For convenience, this method can be called as `response.css()`
##### jmespath(query: str, \*\*kwargs: Any) → SelectorList[Self]
Find objects matching the JMESPath `query` and return the result as a
`SelectorList` instance with all elements flattened. List
elements implement `Selector` interface too.
`query` is a string containing the [JMESPath](https://jmespath.org/) query to apply.
Any additional named arguments are passed to the underlying
`jmespath.search` call, e.g.:
```
selector.jmespath('author.name', options=jmespath.Options(dict_cls=collections.OrderedDict))
```
> [!NOTE]
> For convenience, this method can be called as `response.jmespath()`
##### get() → Any
Serialize and return the matched nodes.
For HTML and XML, the result is always a string, and percent-encoded
content is unquoted.
See also: old-extraction-api
##### attrib
Return the attributes dictionary for underlying element.
For JSON selectors, return an empty dict.
See also: selecting-attributes.
##### re(regex: str | Pattern[str], replace_entities: bool = True) → list[str]
Apply the given regex and return a list of strings with the
matches.
`regex` can be either a compiled regular expression or a string which
will be compiled to a regular expression using `re.compile(regex)`.
By default, character entity references are replaced by their
corresponding character (except for `&` and `<`).
Passing `replace_entities` as `False` switches off these
replacements.
##### re_first(regex: str | Pattern[str], default: None = None, replace_entities: bool = True) → str | None
##### re_first(regex: str | Pattern[str], default: str, replace_entities: bool = True) → str
Apply the given regex and return the first string which matches. If
there is no match, return the default value (`None` if the argument
is not provided).
By default, character entity references are replaced by their
corresponding character (except for `&` and `<`).
Passing `replace_entities` as `False` switches off these
replacements.
##### register_namespace(prefix: str, uri: str) → None
Register the given namespace to be used in this `Selector`.
Without registering namespaces you can’t select or extract data from
non-standard namespaces. See selector-examples-xml.
##### remove_namespaces() → None
Remove all namespaces, allowing to traverse the document using
namespace-less xpaths. See removing-namespaces.
For JSON selectors, this method does nothing.
##### \_\_bool_\_() → bool
Return `True` if there is any real content selected or `False`
otherwise. In other words, the boolean value of a `Selector` is
given by the contents it selects.
##### getall() → list[str]
Serialize and return the matched node in a 1-element list of strings.
This method is added to Selector for consistency; it is more useful
with SelectorList. See also: old-extraction-api
#### SelectorList objects
#### *class* scrapy.selector.SelectorList(iterable=(), /)
The `SelectorList` class is a subclass of the builtin `list`
class, which provides a few additional methods.
##### xpath(xpath: str, namespaces: Mapping[str, str] | None = None, \*\*kwargs: Any) → SelectorList[\_SelectorType]
Call the `.xpath()` method for each element in this list and return
their results flattened as another `SelectorList`.
`xpath` is the same argument as the one in `Selector.xpath()`
`namespaces` is an optional `prefix: namespace-uri` mapping (dict)
for additional prefixes to those registered with `register_namespace(prefix, uri)`.
Contrary to `register_namespace()`, these prefixes are not
saved for future calls.
Any additional named arguments can be used to pass values for XPath
variables in the XPath expression, e.g.:
```
selector.xpath('//a[href=$url]', url="http://www.example.com")
```
##### css(query: str) → SelectorList[\_SelectorType]
Call the `.css()` method for each element in this list and return
their results flattened as another `SelectorList`.
`query` is the same argument as the one in `Selector.css()`
##### jmespath(query: str, \*\*kwargs: Any) → SelectorList[\_SelectorType]
Call the `.jmespath()` method for each element in this list and return
their results flattened as another `SelectorList`.
`query` is the same argument as the one in `Selector.jmespath()`.
Any additional named arguments are passed to the underlying
`jmespath.search` call, e.g.:
```
selector.jmespath('author.name', options=jmespath.Options(dict_cls=collections.OrderedDict))
```
##### getall() → list[str]
Call the `.get()` method for each element is this list and return
their results flattened, as a list of strings.
See also: old-extraction-api
##### get(default: None = None) → str | None
##### get(default: str) → str
Return the result of `.get()` for the first element in this list.
If the list is empty, return the default value.
See also: old-extraction-api
##### re(regex: str | Pattern[str], replace_entities: bool = True) → list[str]
Call the `.re()` method for each element in this list and return
their results flattened, as a list of strings.
By default, character entity references are replaced by their
corresponding character (except for `&` and `<`.
Passing `replace_entities` as `False` switches off these
replacements.
##### re_first(regex: str | Pattern[str], default: None = None, replace_entities: bool = True) → str | None
##### re_first(regex: str | Pattern[str], default: str, replace_entities: bool = True) → str
Call the `.re()` method for the first element in this list and
return the result in an string. If the list is empty or the
regex doesn’t match anything, return the default value (`None` if
the argument is not provided).
By default, character entity references are replaced by their
corresponding character (except for `&` and `<`.
Passing `replace_entities` as `False` switches off these
replacements.
##### attrib
Return the attributes dictionary for the first element.
If the list is empty, return an empty dict.
See also: selecting-attributes.
### Examples
#### Selector examples on HTML response
Here are some `Selector` examples to illustrate several concepts.
In all cases, we assume there is already a `Selector` instantiated with
a `HtmlResponse` object like this:
```python
sel = Selector(html_response)
```
1. Select all `<h1>` elements from an HTML response body, returning a list of
`Selector` objects (i.e. a `SelectorList` object):
```python
sel.xpath("//h1")
```
2. Extract the text of all `<h1>` elements from an HTML response body,
returning a list of strings:
```python
sel.xpath("//h1").getall() # this includes the h1 tag
sel.xpath("//h1/text()").getall() # this excludes the h1 tag
```
3. Iterate over all `<p>` tags and print their class attribute:
```python
for node in sel.xpath("//p"):
print(node.attrib["class"])
```
#### Selector examples on XML response
Here are some examples to illustrate concepts for `Selector` objects
instantiated with an `XmlResponse` object:
```python
sel = Selector(xml_response)
```
1. Select all `<product>` elements from an XML response body, returning a list
of `Selector` objects (i.e. a `SelectorList` object):
```python
sel.xpath("//product")
```
2. Extract all prices from a [Google Base XML feed](https://support.google.com/merchants/answer/14987622) which requires registering
a namespace:
```python
sel.register_namespace("g", "http://base.google.com/ns/1.0")
sel.xpath("//g:price").getall()
```
Source: /topics/items.md
## Items
The main goal in scraping is to extract structured data from unstructured
sources, typically, web pages. Spiders may return the
extracted data as items, Python objects that define key-value pairs.
Scrapy supports multiple types of items. When you create an
item, you may use whichever type of item you want. When you write code that
receives an item, your code should work for any item type.
### Item Types
Scrapy supports the following types of items, via the [itemadapter](https://github.com/scrapy/itemadapter) library:
dictionaries, Item objects,
dataclass objects, attrs objects
and Pydantic models.
#### Dictionaries
As an item type, `dict` is convenient and familiar.
#### Item objects
`Item` provides a `dict`-like API plus additional features that
make it the most feature-complete item type:
#### *class* scrapy.Item(\*args: Any, \*\*kwargs: Any)
Base class for scraped items.
In Scrapy, an object is considered an `item` if it’s supported by the
[itemadapter](https://github.com/scrapy/itemadapter) library. For example, when the output of a spider callback
is evaluated, only such objects are passed to item pipelines. `Item` is one of the classes supported by
[itemadapter](https://github.com/scrapy/itemadapter) by default.
Items must declare `Field` attributes, which are processed and stored
in the `fields` attribute. This restricts the set of allowed field names
and prevents typos, raising `KeyError` when referring to undefined fields.
Additionally, fields can be used to define metadata and control the way
data is processed internally. Please refer to the documentation
about fields for additional information.
Unlike instances of `dict`, instances of `Item` may be
tracked to debug memory leaks.
##### fields *: dict[str, Field]* *= {}*
A dictionary containing *all declared fields* for this Item, not only
those populated. The keys are the field names and the values are the
`Field` objects used in the Item declaration.
Fields are kept in definition order: fields declared in base classes
come first, followed by fields declared in subclasses, and a field
redefined in a subclass keeps the position of its first definition.
> [!WARNING]
> Changed in version 2.17.0: Fields are now returned in definition order rather than alphabetical
> order.
##### copy() → Self
##### deepcopy() → Self
Return a `deepcopy()` of this item.
`Item` objects replicate the standard `dict` API, including
its `__init__` method.
`Item` allows the defining of field names, so that:
- `KeyError` is raised when using undefined field names (i.e.
prevents typos going unnoticed)
- Item exporters can export all fields by
default even if the first scraped object does not have values for all
of them
`Item` also allows the defining of field metadata, which can be used to
customize serialization.
`scrapy.utils.trackref` tracks `Item` objects to help find memory
leaks (see topics-leaks-trackrefs).
Example:
```python
from scrapy.item import Item, Field
class CustomItem(Item):
one_field = Field()
another_field = Field()
```
#### Dataclass objects
`dataclass()` allows the defining of item classes with field names,
so that item exporters can export all fields by
default even if the first scraped object does not have values for all of them.
Additionally, `dataclass` items also allow you to:
* define the type and default value of each defined field.
* define custom field metadata through `dataclasses.field()`, which can be used to
customize serialization.
Example:
```python
from dataclasses import dataclass
@dataclass
class CustomItem:
one_field: str
another_field: int
```
> [!NOTE]
> Field types are not enforced at run time.
#### attr.s objects
`attr.s()` allows the defining of item classes with field names,
so that item exporters can export all fields by
default even if the first scraped object does not have values for all of them.
Additionally, `attr.s` items also allow to:
* define the type and default value of each defined field.
* define custom field metadata, which can be used to
customize serialization.
In order to use this type, the attrs package needs to be installed.
Example:
```python
import attr
@attr.s
class CustomItem:
one_field = attr.ib()
another_field = attr.ib()
```
#### Pydantic models
[Pydantic](https://docs.pydantic.dev/) models allow the defining of item
classes with field names, so that item exporters can
export all fields by default even if the first scraped object does not have
values for all of them.
Additionally, `pydantic` items also allow you to:
* define the type and default value of each defined field with run-time type
validation.
* define custom field metadata through [pydantic.Field](https://docs.pydantic.dev/latest/concepts/fields/), which can be used to
customize serialization.
* benefit from automatic data validation and conversion based on type
annotations.
In order to use this type, the [pydantic package](https://docs.pydantic.dev/)
needs to be installed.
Example:
```python
from pydantic import BaseModel, Field
class CustomItem(BaseModel):
one_field: str = Field(default="", description="First field")
another_field: int = Field(default=0, description="Second field")
```
> [!NOTE]
> Unlike other item types, Pydantic models enforce field types at
> run time and will raise validation errors for invalid data types.
### Working with Item objects
#### Declaring Item subclasses
Item subclasses are declared using a simple class definition syntax and
`Field` objects. Here is an example:
```python
import scrapy
class Product(scrapy.Item):
name = scrapy.Field()
price = scrapy.Field()
stock = scrapy.Field()
tags = scrapy.Field()
last_updated = scrapy.Field(serializer=str)
```
> [!NOTE]
> Those familiar with [Django](https://www.djangoproject.com/) will notice that Scrapy Items are
> declared similar to [Django Models](https://docs.djangoproject.com/en/dev/topics/db/models/), except that Scrapy Items are much
> simpler as there is no concept of different field types.
#### Declaring fields
`Field` objects are used to specify metadata for each field. For
example, the serializer function for the `last_updated` field illustrated in
the example above.
You can specify any kind of metadata for each field. There is no restriction on
the values accepted by `Field` objects. For this same
reason, there is no reference list of all available metadata keys. Each key
defined in `Field` objects could be used by a different component, and
only those components know about it. You can also define and use any other
`Field` key in your project too, for your own needs. The main goal of
`Field` objects is to provide a way to define all field metadata in one
place. Typically, those components whose behaviour depends on each field use
certain field keys to configure that behaviour. You must refer to their
documentation to see which metadata keys are used by each component.
It’s important to note that the `Field` objects used to declare the item
do not stay assigned as class attributes. Instead, they can be accessed through
the `fields` attribute.
#### *class* scrapy.Field
Container of field metadata
The `Field` class is just an alias to the built-in `dict` class and
doesn’t provide any extra functionality or attributes. In other words,
`Field` objects are plain-old Python dicts. A separate class is used
to support the item declaration syntax
based on class attributes.
> [!NOTE]
> Field metadata can also be declared for `dataclass` and `attrs`
> items. Please refer to the documentation for [dataclasses.field](https://docs.python.org/3/library/dataclasses.html#dataclasses.field) and
> [attr.ib](https://www.attrs.org/en/stable/api-attr.html#attr.ib) for additional information.
#### Working with Item objects
Here are some examples of common tasks performed with items, using the
`Product` item declared above. You will
notice the API is very similar to the `dict` API.
##### Creating items
```pycon
>>> product = Product(name="Desktop PC", price=1000)
>>> print(product)
{'name': 'Desktop PC', 'price': 1000}
```
##### Getting field values
```pycon
>>> product["name"]
Desktop PC
>>> product.get("name")
Desktop PC
>>> product["price"]
1000
>>> product["last_updated"]
Traceback (most recent call last):
...
KeyError: 'last_updated'
>>> product.get("last_updated", "not set")
not set
>>> product["lala"] # getting unknown field
Traceback (most recent call last):
...
KeyError: 'lala'
>>> product.get("lala", "unknown field")
'unknown field'
>>> "name" in product # is name field populated?
True
>>> "last_updated" in product # is last_updated populated?
False
>>> "last_updated" in product.fields # is last_updated a declared field?
True
>>> "lala" in product.fields # is lala a declared field?
False
```
##### Setting field values
```pycon
>>> product["last_updated"] = "today"
>>> product["last_updated"]
today
>>> product["lala"] = "test" # setting unknown field
Traceback (most recent call last):
...
KeyError: 'Product does not support field: lala'
```
##### Accessing all populated values
To access all populated values, just use the typical `dict` API:
```pycon
>>> product.keys()
['price', 'name']
>>> product.items()
[('price', 1000), ('name', 'Desktop PC')]
```
##### Copying items
To copy an item, you must first decide whether you want a shallow copy or a
deep copy.
If your item contains mutable values like lists or dictionaries,
a shallow copy will keep references to the same mutable values across all
different copies.
For example, if you have an item with a list of tags, and you create a shallow
copy of that item, both the original item and the copy have the same list of
tags. Adding a tag to the list of one of the items will add the tag to the
other item as well.
If that is not the desired behavior, use a deep copy instead.
See `copy` for more information.
To create a shallow copy of an item, you can either call
`copy()` on an existing item
(`product2 = product.copy()`) or instantiate your item class from an existing
item (`product2 = Product(product)`).
To create a deep copy, call `deepcopy()` instead
(`product2 = product.deepcopy()`).
##### Other common tasks
Creating dicts from items:
```pycon
>>> dict(product) # create a dict from all populated values
{'price': 1000, 'name': 'Desktop PC'}
```
Creating items from dicts:
```pycon
>>> Product({"name": "Laptop PC", "price": 1500})
{'name': 'Laptop PC', 'price': 1500}
>>> Product({"name": "Laptop PC", "lala": 1500}) # warning: unknown field in dict
Traceback (most recent call last):
...
KeyError: 'Product does not support field: lala'
```
#### Extending Item subclasses
You can extend Items (to add more fields or to change some metadata for some
fields) by declaring a subclass of your original Item.
For example:
```python
class DiscountedProduct(Product):
discount_percent = scrapy.Field(serializer=str)
discount_expiration_date = scrapy.Field()
```
You can also extend field metadata by using the previous field metadata and
appending more values, or changing existing values, like this:
```python
class SpecificProduct(Product):
name = scrapy.Field(Product.fields["name"], serializer=my_serializer)
```
That adds (or replaces) the `serializer` metadata key for the `name` field,
keeping all the previously existing metadata values.
### Supporting All Item Types
In code that receives an item, such as methods of item pipelines or spider middlewares, it is a good practice to use the
`ItemAdapter` class to write code that works for any
supported item type.
### Other classes related to items
#### *class* scrapy.item.ItemMeta(class_name: str, bases: tuple[type, ...], attrs: dict[str, Any])
[Metaclass](https://realpython.com/python-metaclasses) of `Item` that handles field definitions.
Source: /topics/loaders.md
## Item Loaders
Item Loaders provide a convenient mechanism for populating scraped items. Even though items can be populated directly, Item Loaders provide a
much more convenient API for populating them from a scraping process, by automating
some common tasks like parsing the raw extracted data before assigning it.
In other words, items provide the *container* of
scraped data, while Item Loaders provide the mechanism for *populating* that
container.
Item Loaders are designed to provide a flexible, efficient and easy mechanism
for extending and overriding different field parsing rules, either by spider,
or by source format (HTML, XML, etc) without becoming a nightmare to maintain.
> [!NOTE]
> Item Loaders are an extension of the [itemloaders](https://itemloaders.readthedocs.io/en/latest/) library that make it
> easier to work with Scrapy by adding support for
> responses.
### Using Item Loaders to populate items
To use an Item Loader, you must first instantiate it. You can either
instantiate it with an item object or without one, in which
case an item object is automatically created in the
Item Loader `__init__` method using the item class
specified in the `ItemLoader.default_item_class` attribute.
Then, you start collecting values into the Item Loader, typically using
Selectors. You can add more than one value to
the same item field; the Item Loader will know how to “join” those values later
using a proper processing function.
> [!NOTE]
> Collected data is internally stored as lists,
> allowing to add several values to the same field.
> If an `item` argument is passed when creating a loader,
> each of the item’s values will be stored as-is if it’s already
> an iterable, or wrapped with a list if it’s a single value.
Here is a typical Item Loader usage in a Spider, using
the Product item declared in the Items
chapter:
```python
from scrapy.loader import ItemLoader
from myproject.items import Product
def parse(self, response):
l = ItemLoader(item=Product(), response=response)
l.add_xpath("name", '//div[@class="product_name"]')
l.add_xpath("name", '//div[@class="product_title"]')
l.add_xpath("price", '//p[@id="price"]')
l.add_css("stock", "p#stock")
l.add_value("last_updated", "today") # you can also use literal values
return l.load_item()
```
By quickly looking at that code, we can see the `name` field is being
extracted from two different XPath locations in the page:
1. `//div[@class="product_name"]`
2. `//div[@class="product_title"]`
In other words, data is being collected by extracting it from two XPath
locations, using the `add_xpath()` method. This is the
data that will be assigned to the `name` field later.
Afterwards, similar calls are used for `price` and `stock` fields
(the latter using a CSS selector with the `add_css()` method),
and finally the `last_updated` field is populated directly with a literal value
(`today`) using a different method: `add_value()`.
Finally, when all data is collected, the `ItemLoader.load_item()` method is
called which actually returns the item populated with the data
previously extracted and collected with the `add_xpath()`,
`add_css()`, and `add_value()` calls.
### Working with dataclass items
By default, dataclass items require all fields to be
passed when created. This could be an issue when using dataclass items with
item loaders: unless a pre-populated item is passed to the loader, fields
will be populated incrementally using the loader’s `add_xpath()`,
`add_css()` and `add_value()` methods.
One approach to overcome this is to define items using the
`field()` function, with a `default` argument:
```python
from dataclasses import dataclass, field
@dataclass
class InventoryItem:
name: str | None = field(default=None)
price: float | None = field(default=None)
stock: int | None = field(default=None)
```
### Input and Output processors
An Item Loader contains one input processor and one output processor for each
(item) field. The input processor processes the extracted data as soon as it’s
received (through the `add_xpath()`, `add_css()` or
`add_value()` methods) and the result of the input processor is
collected and kept inside the ItemLoader. After collecting all data, the
`ItemLoader.load_item()` method is called to populate and get the populated
item object. That’s when the output processor is
called with the data previously collected (and processed using the input
processor). The result of the output processor is the final value that gets
assigned to the item.
Let’s see an example to illustrate how the input and output processors are
called for a particular field (the same applies for any other field):
```python
l = ItemLoader(Product(), some_selector)
l.add_xpath("name", xpath1) # (1)
l.add_xpath("name", xpath2) # (2)
l.add_css("name", css) # (3)
l.add_value("name", "test") # (4)
return l.load_item() # (5)
```
So what happens is:
1. Data from `xpath1` is extracted, and passed through the *input processor* of
the `name` field. The result of the input processor is collected and kept in
the Item Loader (but not yet assigned to the item).
2. Data from `xpath2` is extracted, and passed through the same *input
processor* used in (1). The result of the input processor is appended to the
data collected in (1) (if any).
3. This case is similar to the previous ones, except that the data is extracted
from the `css` CSS selector, and passed through the same *input
processor* used in (1) and (2). The result of the input processor is appended to the
data collected in (1) and (2) (if any).
4. This case is also similar to the previous ones, except that the value to be
collected is assigned directly, instead of being extracted from a XPath
expression or a CSS selector.
However, the value is still passed through the input processors. In this
case, since the value is not iterable it is converted to an iterable of a
single element before passing it to the input processor, because input
processor always receive iterables.
5. The data collected in steps (1), (2), (3) and (4) is passed through
the *output processor* of the `name` field.
The result of the output processor is the value assigned to the `name`
field in the item.
It’s worth noticing that processors are just callable objects, which are called
with the data to be parsed, and return a parsed value. So you can use any
function as input or output processor. The only requirement is that they must
accept one (and only one) positional argument, which will be an iterable.
> [!NOTE]
> Both input and output processors must receive an iterable as their
> first argument. The output of those functions can be anything. The result of
> input processors will be appended to an internal list (in the Loader)
> containing the collected values (for that field). The result of the output
> processors is the value that will be finally assigned to the item.
The other thing you need to keep in mind is that the values returned by input
processors are collected internally (in lists) and then passed to output
processors to populate the fields.
Last, but not least, [itemloaders](https://itemloaders.readthedocs.io/en/latest/) comes with some commonly used
processors built-in for convenience.
### Declaring Item Loaders
Item Loaders are declared using a class definition syntax. Here is an example:
```python
from itemloaders.processors import TakeFirst, MapCompose, Join
from scrapy.loader import ItemLoader
class ProductLoader(ItemLoader):
default_output_processor = TakeFirst()
name_in = MapCompose(str.title)
name_out = Join()
price_in = MapCompose(str.strip)
# ...
```
As you can see, input processors are declared using the `_in` suffix while
output processors are declared using the `_out` suffix. And you can also
declare a default input/output processors using the
`ItemLoader.default_input_processor` and
`ItemLoader.default_output_processor` attributes.
### Declaring Input and Output Processors
As seen in the previous section, input and output processors can be declared in
the Item Loader definition, and it’s very common to declare input processors
this way. However, there is one more place where you can specify the input and
output processors to use: in the Item Field
metadata. Here is an example:
```python
from dataclasses import dataclass, field
from itemloaders.processors import Join, MapCompose, TakeFirst
from w3lib.html import remove_tags
def filter_price(value):
if value.isdigit():
return value
@dataclass
class Product:
name: str | None = field(
default=None,
metadata={
"input_processor": MapCompose(remove_tags),
"output_processor": Join(),
},
)
price: str | None = field(
default=None,
metadata={
"input_processor": MapCompose(remove_tags, filter_price),
"output_processor": TakeFirst(),
},
)
```
```pycon
>>> from scrapy.loader import ItemLoader
>>> il = ItemLoader(item=Product())
>>> il.add_value("name", ["Welcome to my", "<strong>website</strong>"])
>>> il.add_value("price", ["€", "<span>1000</span>"])
>>> il.load_item()
Product(name='Welcome to my website', price='1000')
```
The precedence order, for both input and output processors, is as follows:
1. Item Loader field-specific attributes: `field_in` and `field_out` (most
precedence)
2. Field metadata (`input_processor` and `output_processor` key)
3. Item Loader defaults: `ItemLoader.default_input_processor` and
`ItemLoader.default_output_processor` (least precedence)
See also: topics-loaders-extending.
### Item Loader Context
The Item Loader Context is a dict of arbitrary key/values which is shared among
all input and output processors in the Item Loader. It can be passed when
declaring, instantiating or using Item Loader. They are used to modify the
behaviour of the input/output processors.
For example, suppose you have a function `parse_length` which receives a text
value and extracts a length from it:
```python
def parse_length(text, loader_context):
unit = loader_context.get("unit", "m")
# ... length parsing code goes here ...
return parsed_length
```
By accepting a `loader_context` argument the function is explicitly telling
the Item Loader that it’s able to receive an Item Loader context, so the Item
Loader passes the currently active context when calling it, and the processor
function (`parse_length` in this case) can thus use them.
There are several ways to modify Item Loader context values:
1. By modifying the currently active Item Loader context
(`context` attribute):
```python
loader = ItemLoader(product)
loader.context["unit"] = "cm"
```
2. On Item Loader instantiation (the keyword arguments of Item Loader
`__init__` method are stored in the Item Loader context):
```python
loader = ItemLoader(product, unit="cm")
```
3. On Item Loader declaration, for those input/output processors that support
instantiating them with an Item Loader context.
`MapCompose` is one of them:
```python
class ProductLoader(ItemLoader):
length_out = MapCompose(parse_length, unit="cm")
```
### ItemLoader objects
#### *class* scrapy.loader.ItemLoader(item: Any = None, selector: Selector | None = None, response: TextResponse | None = None, parent: itemloaders.ItemLoader | None = None, \*\*context: Any)
A user-friendly abstraction to populate an item with data
by applying field processors to scraped data.
When instantiated with a `selector` or a `response` it supports
data extraction from web pages using selectors.
* **Parameters:**
* **item** (item object) – The item instance to populate using subsequent calls to
`add_xpath()`, `add_css()`,
or `add_value()`.
* **selector** (`Selector` object) – The selector to extract data from, when using the
`add_xpath()`, `add_css()`, `replace_xpath()`, or
`replace_css()` method.
* **response** (`Response` object) – The response used to construct the selector using the
`default_selector_class`, unless the selector argument is given,
in which case this argument is ignored.
If no item is given, one is instantiated automatically using the class in
`default_item_class`.
The item, selector, response and remaining keyword arguments are
assigned to the Loader context (accessible through the `context` attribute).
##### item
The item object being parsed by this Item Loader.
This is mostly used as a property so, when attempting to override this
value, you may want to check out `default_item_class` first.
##### context
The currently active Context of this Item Loader.
##### default_item_class
An item class (or factory), used to instantiate
items when not given in the `__init__` method.
##### default_input_processor
The default input processor to use for those fields which don’t specify
one.
##### default_output_processor
The default output processor to use for those fields which don’t specify
one.
##### default_selector_class
The class used to construct the `selector` of this
`ItemLoader`, if only a response is given in the `__init__` method.
If a selector is given in the `__init__` method this attribute is ignored.
This attribute is sometimes overridden in subclasses.
##### selector
The `Selector` object to extract data from.
It’s either the selector given in the `__init__` method or one created from
the response given in the `__init__` method using the
`default_selector_class`. This attribute is meant to be
read-only.
##### add_css(field_name: str | None, css: str | Iterable[str], \*processors: Callable[..., Any], re: str | Pattern[str] | None = None, \*\*kw: Any) → Self
Similar to `ItemLoader.add_value()` but receives a CSS selector
instead of a value, which is used to extract a list of unicode strings
from the selector associated with this `ItemLoader`.
See `get_css()` for `kwargs`.
* **Parameters:**
**css** (*str*) – the CSS selector to extract data from
* **Returns:**
The current ItemLoader instance for method chaining.
* **Return type:**
ItemLoader
Examples:
```
# HTML snippet: <p class="product-name">Color TV</p>
loader.add_css('name', 'p.product-name')
# HTML snippet: <p id="price">the price is $1200</p>
loader.add_css('price', 'p#price', re='the price is (.*)')
```
##### add_jmes(field_name: str | None, jmes: str, \*processors: Callable[..., Any], re: str | Pattern[str] | None = None, \*\*kw: Any) → Self
Similar to `ItemLoader.add_value()` but receives a JMESPath selector
instead of a value, which is used to extract a list of unicode strings
from the selector associated with this `ItemLoader`.
See `get_jmes()` for `kwargs`.
* **Parameters:**
**jmes** (*str*) – the JMESPath selector to extract data from
* **Returns:**
The current ItemLoader instance for method chaining.
* **Return type:**
ItemLoader
Examples:
```
# HTML snippet: {"name": "Color TV"}
loader.add_jmes('name')
# HTML snippet: {"price": the price is $1200"}
loader.add_jmes('price', TakeFirst(), re='the price is (.*)')
```
##### add_value(field_name: str | None, value: Any, \*processors: Callable[..., Any], re: str | Pattern[str] | None = None, \*\*kw: Any) → Self
Process and then add the given `value` for the given field.
The value is first passed through `get_value()` by giving the
`processors` and `kwargs`, and then passed through the
field input processor and its result
appended to the data collected for that field. If the field already
contains collected data, the new data is added.
The given `field_name` can be `None`, in which case values for
multiple fields may be added. And the processed value should be a dict
with field_name mapped to values.
* **Returns:**
The current ItemLoader instance for method chaining.
* **Return type:**
ItemLoader
Examples:
```
loader.add_value('name', 'Color TV')
loader.add_value('colours', ['white', 'blue'])
loader.add_value('length', '100')
loader.add_value('name', 'name: foo', TakeFirst(), re='name: (.+)')
loader.add_value(None, {'name': 'foo', 'sex': 'male'})
```
##### add_xpath(field_name: str | None, xpath: str | Iterable[str], \*processors: Callable[..., Any], re: str | Pattern[str] | None = None, \*\*kw: Any) → Self
Similar to `ItemLoader.add_value()` but receives an XPath instead of a
value, which is used to extract a list of strings from the
selector associated with this `ItemLoader`.
See `get_xpath()` for `kwargs`.
* **Parameters:**
**xpath** (*str*) – the XPath to extract data from
* **Returns:**
The current ItemLoader instance for method chaining.
* **Return type:**
ItemLoader
Examples:
```
# HTML snippet: <p class="product-name">Color TV</p>
loader.add_xpath('name', '//p[@class="product-name"]')
# HTML snippet: <p id="price">the price is $1200</p>
loader.add_xpath('price', '//p[@id="price"]', re='the price is (.*)')
```
##### get_collected_values(field_name: str) → list[Any]
Return the collected values for the given field.
##### get_css(css: str | Iterable[str], \*processors: Callable[..., Any], re: str | Pattern[str] | None = None, \*\*kw: Any) → Any
Similar to `ItemLoader.get_value()` but receives a CSS selector
instead of a value, which is used to extract a list of unicode strings
from the selector associated with this `ItemLoader`.
* **Parameters:**
* **css** (*str*) – the CSS selector to extract data from
* **re** (*str* *or* *Pattern* *[**str* *]*) – a regular expression to use for extracting data from the
selected CSS region
Examples:
```
# HTML snippet: <p class="product-name">Color TV</p>
loader.get_css('p.product-name')
# HTML snippet: <p id="price">the price is $1200</p>
loader.get_css('p#price', TakeFirst(), re='the price is (.*)')
```
##### get_jmes(jmes: str | Iterable[str], \*processors: Callable[..., Any], re: str | Pattern[str] | None = None, \*\*kw: Any) → Any
Similar to `ItemLoader.get_value()` but receives a JMESPath selector
instead of a value, which is used to extract a list of unicode strings
from the selector associated with this `ItemLoader`.
* **Parameters:**
* **jmes** (*str*) – the JMESPath selector to extract data from
* **re** (*str* *or* *Pattern*) – a regular expression to use for extracting data from the
selected JMESPath
Examples:
```
# HTML snippet: {"name": "Color TV"}
loader.get_jmes('name')
# HTML snippet: {"price": the price is $1200"}
loader.get_jmes('price', TakeFirst(), re='the price is (.*)')
```
##### get_output_value(field_name: str) → Any
Return the collected values parsed using the output processor, for the
given field. This method doesn’t populate or modify the item at all.
##### get_value(value: Any, \*processors: Callable[..., Any], re: str | Pattern[str] | None = None, \*\*kw: Any) → Any
Process the given `value` by the given `processors` and keyword
arguments.
Available keyword arguments:
* **Parameters:**
**re** (*str* *or* *Pattern* *[**str* *]*) – a regular expression to use for extracting data from the
given value using `extract_regex()` method,
applied before processors
Examples:
```pycon
>>> from itemloaders import ItemLoader
>>> from itemloaders.processors import TakeFirst
>>> loader = ItemLoader()
>>> loader.get_value('name: foo', TakeFirst(), str.upper, re='name: (.+)')
'FOO'
```
##### get_xpath(xpath: str | Iterable[str], \*processors: Callable[..., Any], re: str | Pattern[str] | None = None, \*\*kw: Any) → Any
Similar to `ItemLoader.get_value()` but receives an XPath instead of a
value, which is used to extract a list of unicode strings from the
selector associated with this `ItemLoader`.
* **Parameters:**
* **xpath** (*str*) – the XPath to extract data from
* **re** (*str* *or* *Pattern* *[**str* *]*) – a regular expression to use for extracting data from the
selected XPath region
Examples:
```
# HTML snippet: <p class="product-name">Color TV</p>
loader.get_xpath('//p[@class="product-name"]')
# HTML snippet: <p id="price">the price is $1200</p>
loader.get_xpath('//p[@id="price"]', TakeFirst(), re='the price is (.*)')
```
##### load_item() → Any
Populate the item with the data collected so far, and return it. The
data collected is first passed through the output processors to get the final value to assign to each item field.
##### nested_css(css: str, \*\*context: Any) → Self
Create a nested loader with a css selector.
The supplied selector is applied relative to selector associated
with this `ItemLoader`. The nested loader shares the item
with the parent `ItemLoader` so calls to `add_xpath()`,
`add_value()`, `replace_value()`, etc. will behave as expected.
##### nested_xpath(xpath: str, \*\*context: Any) → Self
Create a nested loader with an xpath selector.
The supplied selector is applied relative to selector associated
with this `ItemLoader`. The nested loader shares the item
with the parent `ItemLoader` so calls to `add_xpath()`,
`add_value()`, `replace_value()`, etc. will behave as expected.
##### replace_css(field_name: str | None, css: str | Iterable[str], \*processors: Callable[..., Any], re: str | Pattern[str] | None = None, \*\*kw: Any) → Self
Similar to `add_css()` but replaces collected data instead of adding it.
* **Returns:**
The current ItemLoader instance for method chaining.
* **Return type:**
ItemLoader
##### replace_jmes(field_name: str | None, jmes: str | Iterable[str], \*processors: Callable[..., Any], re: str | Pattern[str] | None = None, \*\*kw: Any) → Self
Similar to `add_jmes()` but replaces collected data instead of adding it.
* **Returns:**
The current ItemLoader instance for method chaining.
* **Return type:**
ItemLoader
##### replace_value(field_name: str | None, value: Any, \*processors: Callable[..., Any], re: str | Pattern[str] | None = None, \*\*kw: Any) → Self
Similar to `add_value()` but replaces the collected data with the
new value instead of adding it.
* **Returns:**
The current ItemLoader instance for method chaining.
* **Return type:**
ItemLoader
##### replace_xpath(field_name: str | None, xpath: str | Iterable[str], \*processors: Callable[..., Any], re: str | Pattern[str] | None = None, \*\*kw: Any) → Self
Similar to `add_xpath()` but replaces collected data instead of adding it.
* **Returns:**
The current ItemLoader instance for method chaining.
* **Return type:**
ItemLoader
### Nested Loaders
When parsing related values from a subsection of a document, it can be
useful to create nested loaders. Imagine you’re extracting details from
a footer of a page that looks something like:
Example:
```html
<footer>
<a class="social" href="https://facebook.com/whatever">Like Us</a>
<a class="social" href="https://twitter.com/whatever">Follow Us</a>
<a class="email" href="mailto:whatever@example.com">Email Us</a>
</footer>
```
Without nested loaders, you need to specify the full xpath (or css) for each value
that you wish to extract.
Example:
```python
loader = ItemLoader(item=Item())
# load stuff not in the footer
loader.add_xpath("social", '//footer/a[@class = "social"]/@href')
loader.add_xpath("email", '//footer/a[@class = "email"]/@href')
loader.load_item()
```
Instead, you can create a nested loader with the footer selector and add values
relative to the footer. The functionality is the same but you avoid repeating
the footer selector.
Example:
```python
loader = ItemLoader(item=Item())
# load stuff not in the footer
footer_loader = loader.nested_xpath("//footer")
footer_loader.add_xpath("social", 'a[@class = "social"]/@href')
footer_loader.add_xpath("email", 'a[@class = "email"]/@href')
# no need to call footer_loader.load_item()
loader.load_item()
```
You can nest loaders arbitrarily and they work with either xpath or css selectors.
As a general guideline, use nested loaders when they make your code simpler but do
not go overboard with nesting or your parser can become difficult to read.
### Reusing and extending Item Loaders
As your project grows bigger and acquires more and more spiders, maintenance
becomes a fundamental problem, especially when you have to deal with many
different parsing rules for each spider, having a lot of exceptions, but also
wanting to reuse the common processors.
Item Loaders are designed to ease the maintenance burden of parsing rules,
without losing flexibility and, at the same time, providing a convenient
mechanism for extending and overriding them. For this reason Item Loaders
support traditional Python class inheritance for dealing with differences of
specific spiders (or groups of spiders).
Suppose, for example, that some particular site encloses their product names in
three dashes (e.g. `---Plasma TV---`) and you don’t want to end up scraping
those dashes in the final product names.
Here’s how you can remove those dashes by reusing and extending the default
Product Item Loader (`ProductLoader`):
```python
from itemloaders.processors import MapCompose
from myproject.ItemLoaders import ProductLoader
def strip_dashes(x):
return x.strip("-")
class SiteSpecificLoader(ProductLoader):
name_in = MapCompose(strip_dashes, ProductLoader.name_in)
```
Another case where extending Item Loaders can be very helpful is when you have
multiple source formats, for example XML and HTML. In the XML version you may
want to remove `CDATA` occurrences. Here’s an example of how to do it:
```python
from itemloaders.processors import MapCompose
from myproject.ItemLoaders import ProductLoader
from myproject.utils.xml import remove_cdata
class XmlProductLoader(ProductLoader):
name_in = MapCompose(remove_cdata, ProductLoader.name_in)
```
And that’s how you typically extend input processors.
As for output processors, it is more common to declare them in the field metadata,
as they usually depend only on the field and not on each specific site parsing
rule (as input processors do). See also:
topics-loaders-processors-declaring.
There are many other possible ways to extend, inherit and override your Item
Loaders, and different Item Loaders hierarchies may fit better for different
projects. Scrapy only provides the mechanism; it doesn’t impose any specific
organization of your Loaders collection - that’s up to you and your project’s
needs.
Source: /topics/shell.md
## Scrapy shell
The Scrapy shell is an interactive shell where you can try and debug your
scraping code very quickly, without having to run the spider. It’s meant to be
used for testing data extraction code, but you can actually use it for testing
any kind of code as it is also a regular Python shell.
The shell is used for testing XPath or CSS expressions and see how they work
and what data they extract from the web pages you’re trying to scrape. It
allows you to interactively test your expressions while you’re writing your
spider, without having to run the spider to test every change.
Once you get familiarized with the Scrapy shell, you’ll see that it’s an
invaluable tool for developing and debugging your spiders.
### Configuring the shell
With the ptpython extra, the Scrapy shell will use [ptpython](https://github.com/prompt-toolkit/ptpython)
instead of the REPL. ptpython provides syntax highlighting, smart
auto-completion, and more.
Failing that, with the ipython extra, the Scrapy shell will
use [IPython](https://ipython.org/) instead. IPython provides smart auto-completion, colorized
output, and more.
Scrapy also has support for [bpython](https://bpython-interpreter.org/) via the bpython extra,
and will try to use it where neither ptpython nor IPython is available.
Through Scrapy’s settings you can configure it to use any one of
`ptpython`, `ipython`, `bpython` or the standard `python` shell,
regardless of which are installed. This is done by setting the
`SCRAPY_PYTHON_SHELL` environment variable; or by defining it in your
scrapy.cfg:
```ini
[settings]
shell = bpython
```
### Launch the shell
To launch the Scrapy shell you can use the `shell` command like
this:
```
scrapy shell <url>
```
Where the `<url>` is the URL you want to scrape.
`shell` also works for local files. This can be handy if you want
to play around with a local copy of a web page. `shell` understands
the following syntaxes for local files:
```
# UNIX-style
scrapy shell ./path/to/file.html
scrapy shell ../other/path/to/file.html
scrapy shell /absolute/path/to/file.html
# File URI
scrapy shell file:///absolute/path/to/file.html
```
> [!NOTE]
> When using relative file paths, be explicit and prepend them
> with `./` (or `../` when relevant).
> `scrapy shell index.html` will not work as one might expect (and
> this is by design, not a bug).
>
> Because `shell` favors HTTP URLs over File URIs,
> and `index.html` being syntactically similar to `example.com`,
> `shell` will treat `index.html` as a domain name and trigger
> a DNS lookup error:
>
> ```
> $ scrapy shell index.html
> [ ... scrapy shell starts ... ]
> [ ... traceback ... ]
> twisted.internet.error.DNSLookupError: DNS lookup failed:
> address 'index.html' not found: [Errno -5] No address associated with hostname.
> ```
>
> `shell` will not test beforehand if a file called `index.html`
> exists in the current directory. Again, be explicit.
### Using the shell
The Scrapy shell is just a regular Python console (or [IPython](https://ipython.org/) console if you
have it available) which provides some additional shortcut functions for
convenience.
#### Available Shortcuts
- `shelp()` - print a help with the list of available objects and
shortcuts
- `fetch(url[, redirect=True])` - fetch a new response from the given URL
and update all related objects accordingly. You can optionally ask for HTTP
3xx redirections to not be followed by passing `redirect=False`
- `fetch(request)` - fetch a new response from the given request and update
all related objects accordingly.
- `view(response)` - open the given response in your local web browser, for
inspection. This will add a [<base> tag](https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/base) to the response body in order
for external links (such as images and style sheets) to display properly.
Note, however, that this will create a temporary file in your computer,
which won’t be removed automatically.
#### Available Scrapy objects
The Scrapy shell automatically creates some convenient objects from the
downloaded page, like the `Response` object and the
`Selector` objects (for both HTML and XML
content).
Those objects are:
- `crawler` - the current `Crawler` object.
- `spider` - the Spider which is known to handle the URL, or a
`Spider` object if there is no spider found for the
current URL
- `request` - a `Request` object of the last fetched
page. You can modify this request using
`replace()` or fetch a new request (without
leaving the shell) using the `fetch` shortcut.
- `response` - a `Response` object containing the last
fetched page
- `settings` - the current Scrapy settings
#### Adding your own objects
To define additional objects, or to run code every time a response is fetched,
write a custom project command in a module called
`shell`, which overrides the `shell` command, and override its
`update_vars` method. It is called on start and after every `fetch`, and it
receives the mapping of variable names to objects:
```python
from scrapy.commands.shell import Command as ShellCommand
class Command(ShellCommand):
def update_vars(self, vars):
from myproject.utils import parse_product
vars["parse_product"] = parse_product
if vars["response"] is not None:
vars["product"] = parse_product(vars["response"])
```
`response` is `None` when the shell is started without a URL.
### Example of shell session
Here’s an example of a typical shell session where we start by scraping the
[https://www.scrapy.org/](https://www.scrapy.org/) page, and then proceed to scrape the [https://old.reddit.com/](https://old.reddit.com/)
page. Finally, we modify the (Reddit) request method to POST and re-fetch it
getting an error. We end the session by typing Ctrl-D (in Unix systems) or
Ctrl-Z in Windows.
Keep in mind that the data extracted here may not be the same when you try it,
as those pages are not static and could have changed by the time you test this.
The only purpose of this example is to get you familiarized with how the Scrapy
shell works.
First, we launch the shell:
```
scrapy shell 'https://scrapy.org' --nolog
```
> [!NOTE]
> Remember to always enclose URLs in quotes when running the Scrapy shell from
> the command line, otherwise URLs containing arguments (i.e. the `&` character)
> will not work.
>
> On Windows, use double quotes instead:
>
> ```
> scrapy shell "https://scrapy.org" --nolog
> ```
Then, the shell fetches the URL (using the Scrapy downloader) and prints the
list of available objects and useful shortcuts (you’ll notice that these lines
all start with the `[s]` prefix):
```
[s] Available Scrapy objects:
[s] scrapy scrapy module (contains scrapy.Request, scrapy.Selector, etc)
[s] crawler <scrapy.crawler.Crawler object at 0x7f07395dd690>
[s] item {}
[s] request <GET https://scrapy.org>
[s] response <200 https://scrapy.org/>
[s] settings <scrapy.settings.Settings object at 0x7f07395dd710>
[s] spider <DefaultSpider 'default' at 0x7f0735891690>
[s] Useful shortcuts:
[s] fetch(url[, redirect=True]) Fetch URL and update local objects (by default, redirects are followed)
[s] fetch(req) Fetch a scrapy.Request and update local objects
[s] shelp() Shell help (print this help)
[s] view(response) View response in a browser
>>>
```
After that, we can start playing with the objects:
```pycon
>>> response.xpath("//title/text()").get()
'Scrapy | A Fast and Powerful Scraping and Web Crawling Framework'
>>> fetch("https://old.reddit.com/")
>>> response.xpath("//title/text()").get()
'reddit: the front page of the internet'
>>> request = request.replace(method="POST")
>>> fetch(request)
>>> response.status
404
>>> from pprint import pprint
>>> pprint(response.headers)
{'Accept-Ranges': ['bytes'],
'Cache-Control': ['max-age=0, must-revalidate'],
'Content-Type': ['text/html; charset=UTF-8'],
'Date': ['Thu, 08 Dec 2016 16:21:19 GMT'],
'Server': ['snooserv'],
'Set-Cookie': ['loid=KqNLou0V9SKMX4qb4n; Domain=reddit.com; Max-Age=63071999; Path=/; expires=Sat, 08-Dec-2018 16:21:19 GMT; secure',
'loidcreated=2016-12-08T16%3A21%3A19.445Z; Domain=reddit.com; Max-Age=63071999; Path=/; expires=Sat, 08-Dec-2018 16:21:19 GMT; secure',
'loid=vi0ZVe4NkxNWdlH7r7; Domain=reddit.com; Max-Age=63071999; Path=/; expires=Sat, 08-Dec-2018 16:21:19 GMT; secure',
'loidcreated=2016-12-08T16%3A21%3A19.459Z; Domain=reddit.com; Max-Age=63071999; Path=/; expires=Sat, 08-Dec-2018 16:21:19 GMT; secure'],
'Vary': ['accept-encoding'],
'Via': ['1.1 varnish'],
'X-Cache': ['MISS'],
'X-Cache-Hits': ['0'],
'X-Content-Type-Options': ['nosniff'],
'X-Frame-Options': ['SAMEORIGIN'],
'X-Moose': ['majestic'],
'X-Served-By': ['cache-cdg8730-CDG'],
'X-Timer': ['S1481214079.394283,VS0,VE159'],
'X-Ua-Compatible': ['IE=edge'],
'X-Xss-Protection': ['1; mode=block']}
```
### Invoking the shell from spiders to inspect responses
Sometimes you want to inspect the responses that are being processed in a
certain point of your spider, if only to check that response you expect is
getting there.
This can be achieved by using the `scrapy.shell.inspect_response` function.
Here’s an example of how you would call it from your spider:
```python
import scrapy
class MySpider(scrapy.Spider):
name = "myspider"
start_urls = [
"http://example.com",
"http://example.org",
"http://example.net",
]
def parse(self, response):
# We want to inspect one specific response.
if ".org" in response.url:
from scrapy.shell import inspect_response
inspect_response(response, self)
# Rest of parsing code.
```
When you run the spider, you will get something similar to this:
```
2014-01-23 17:48:31-0400 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://example.com> (referer: None)
2014-01-23 17:48:31-0400 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://example.org> (referer: None)
[s] Available Scrapy objects:
[s] crawler <scrapy.crawler.Crawler object at 0x1e16b50>
...
>>> response.url
'http://example.org'
```
Then, you can check if the extraction code is working:
```pycon
>>> response.xpath('//h1[@class="fn"]')
[]
```
Nope, it doesn’t. So you can open the response in your web browser and see if
it’s the response you were expecting:
```pycon
>>> view(response)
True
```
Finally you hit Ctrl-D (or Ctrl-Z in Windows) to exit the shell and resume the
crawling:
```
>>> ^D
2014-01-23 17:50:03-0400 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://example.net> (referer: None)
...
```
Note that you can’t use the `fetch` shortcut here since the Scrapy engine is
blocked by the shell. However, after you leave the shell, the spider will
continue crawling where it stopped, as shown above.
Source: /topics/item-pipeline.md
## Item Pipeline
After an item has been scraped by a spider, it is sent to the Item Pipeline
which processes it through several components that are executed sequentially.
Each item pipeline component (sometimes referred as just “Item Pipeline”) is a
Python class that implements a simple method. They receive an item and perform
an action over it, also deciding if the item should continue through the
pipeline or be dropped and no longer processed.
Typical uses of item pipelines are:
* cleansing HTML data
* validating scraped data (checking that the items contain certain fields)
* checking for duplicates (and dropping them)
* storing the scraped item in a database
### Writing your own item pipeline
Each item pipeline is a component that must
implement the following method:
#### process_item(self, item)
This method is called for every item pipeline component.
item is an item object, see
supporting-item-types.
`process_item()` must either return an item object
or raise a `DropItem` exception.
Dropped items are no longer processed by further pipeline components.
* **Parameters:**
**item** (item object) – the scraped item
Additionally, they may also implement the following methods:
#### open_spider(self)
This method is called when the spider is opened.
> [!WARNING]
> Changed in version 2.18.0: Added support for `CloseSpider`.
It may raise `CloseSpider` to close the spider before
it starts crawling, e.g. if a resource that the pipeline needs is
unavailable.
#### close_spider(self)
This method is called when the spider is closed, before the
`spider_closed` signal is sent.
Any of these methods may be defined as a coroutine function (`async def`).
### Item pipeline example
#### Price validation and dropping items with no prices
Let’s take a look at the following hypothetical pipeline that adjusts the
`price` attribute for those items that do not include VAT
(`price_excludes_vat` attribute), and drops those items which don’t
contain a price:
```python
from itemadapter import ItemAdapter
from scrapy.exceptions import DropItem
class PricePipeline:
vat_factor = 1.15
def process_item(self, item):
adapter = ItemAdapter(item)
if adapter.get("price"):
if adapter.get("price_excludes_vat"):
adapter["price"] = adapter["price"] * self.vat_factor
return item
else:
raise DropItem("Missing price")
```
#### Write items to a JSON lines file
The following pipeline stores all scraped items (from all spiders) into a
single `items.jsonl` file, containing one item per line serialized in JSON
format:
```python
import json
from itemadapter import ItemAdapter
class JsonWriterPipeline:
def open_spider(self):
self.file = open("items.jsonl", "w")
def close_spider(self):
self.file.close()
def process_item(self, item):
line = json.dumps(ItemAdapter(item).asdict()) + "\n"
self.file.write(line)
return item
```
> [!NOTE]
> The purpose of JsonWriterPipeline is just to introduce how to write
> item pipelines. If you really want to store all scraped items into a JSON
> file you should use the Feed exports.
#### Write items to MongoDB
In this example we’ll write items to [MongoDB](https://www.mongodb.com/) using [pymongo](https://pymongo.readthedocs.io/en/stable/).
MongoDB address and database name are specified in Scrapy settings;
MongoDB collection is specified in a class attribute.
The main point of this example is to show how to get the crawler and how to clean up the resources properly.
```python
import pymongo
from itemadapter import ItemAdapter
class MongoPipeline:
collection_name = "scrapy_items"
def __init__(self, mongo_uri, mongo_db):
self.mongo_uri = mongo_uri
self.mongo_db = mongo_db
@classmethod
def from_crawler(cls, crawler):
return cls(
mongo_uri=crawler.settings.get("MONGO_URI"),
mongo_db=crawler.settings.get("MONGO_DATABASE", "items"),
)
def open_spider(self):
self.client = pymongo.MongoClient(self.mongo_uri)
self.db = self.client[self.mongo_db]
def close_spider(self):
self.client.close()
def process_item(self, item):
self.db[self.collection_name].insert_one(ItemAdapter(item).asdict())
return item
```
#### Take screenshot of item
This example demonstrates how to use coroutine syntax in
the `process_item()` method.
This item pipeline makes a request to a locally-running instance of [Splash](https://splash.readthedocs.io/en/stable/) to
render a screenshot of the item URL. After the request response is downloaded,
the item pipeline saves the screenshot to a file and adds the filename to the
item.
```python
import hashlib
from pathlib import Path
from urllib.parse import quote
import scrapy
from itemadapter import ItemAdapter
from scrapy.http.request import NO_CALLBACK
class ScreenshotPipeline:
"""Pipeline that uses Splash to render screenshot of
every Scrapy item."""
SPLASH_URL = "http://localhost:8050/render.png?url={}"
def __init__(self, crawler):
self.crawler = crawler
@classmethod
def from_crawler(cls, crawler):
return cls(crawler)
async def process_item(self, item):
adapter = ItemAdapter(item)
encoded_item_url = quote(adapter["url"])
screenshot_url = self.SPLASH_URL.format(encoded_item_url)
request = scrapy.Request(screenshot_url, callback=NO_CALLBACK)
response = await self.crawler.engine.download_async(request)
if response.status != 200:
# Error happened, return item.
return item
# Save screenshot to file, filename will be hash of url.
url = adapter["url"]
url_hash = hashlib.md5(url.encode("utf8")).hexdigest()
filename = f"{url_hash}.png"
Path(filename).write_bytes(response.body)
# Store filename in item.
adapter["screenshot_filename"] = filename
return item
```
#### Duplicates filter
A filter that looks for duplicate items, and drops those items that were
already processed. Let’s say that our items have a unique id, but our spider
returns multiples items with the same id:
```python
from itemadapter import ItemAdapter
from scrapy.exceptions import DropItem
class DuplicatesPipeline:
def __init__(self):
self.ids_seen = set()
def process_item(self, item):
adapter = ItemAdapter(item)
if adapter["id"] in self.ids_seen:
raise DropItem(f"Item ID already seen: {adapter['id']}")
else:
self.ids_seen.add(adapter["id"])
return item
```
### Activating an Item Pipeline component
To activate an Item Pipeline component you must add its class to the
`ITEM_PIPELINES` setting, like in the following example:
```python
ITEM_PIPELINES = {
"myproject.pipelines.PricePipeline": 300,
"myproject.pipelines.JsonWriterPipeline": 800,
}
```
The integer values you assign to classes in this setting determine the
order in which they run: items go through from lower valued to higher
valued classes. It’s customary to define these numbers in the 0-1000 range.
### A complete example
The examples above show item pipeline components on their own. In a project, a
pipeline is one of four pieces that work together: the item your spider produces, the spider that
yields it, the pipeline that processes it, and the `ITEM_PIPELINES`
setting that enables the pipeline.
The following example wires those pieces together to validate the price of
books scraped from [books.toscrape.com](https://books.toscrape.com/), reusing the `PricePipeline` from
price-pipeline-example above.
Define the item in `myproject/items.py`:
```python
from dataclasses import dataclass
@dataclass
class BookItem:
title: str
price: float
```
Yield instances of that item from your spider, e.g. in
`myproject/spiders/books.py`:
```python
import scrapy
from myproject.items import BookItem
class BooksSpider(scrapy.Spider):
name = "books"
start_urls = ["https://books.toscrape.com/"]
def parse(self, response):
for book in response.css("article.product_pod"):
yield BookItem(
title=book.css("h3 a::attr(title)").get(),
price=float(book.css("p.price_color::text").re_first(r"[\d.]+")),
)
```
Put the `PricePipeline` shown earlier in `myproject/pipelines.py`, and
enable it in `myproject/settings.py`:
```python
ITEM_PIPELINES = {
"myproject.pipelines.PricePipeline": 300,
}
```
With these pieces in place, every `BookItem` that `BooksSpider` yields
passes through `PricePipeline` before it reaches the feed exports or any other output.
### Testing an item pipeline
To send the items from a single URL through your item pipelines, use the
`parse` command with the `--pipelines` option:
```
scrapy parse --pipelines "https://books.toscrape.com/"
```
To test specific item data instead, add a callback that builds an item out of
its keyword arguments:
```python
class BooksSpider(scrapy.Spider):
# ...
def parse_item(self, response, **fields):
yield BookItem(**fields)
```
and pass those keyword arguments in the command line:
```
scrapy parse --pipelines -c parse_item --cbkwargs '{"title": "Test", "price": 10}' "https://books.toscrape.com/"
```
Pass any URL that your spider handles; it is downloaded even though the
callback ignores it.
### Common pitfalls
#### The pipeline does not run
A pipeline component only runs if its class is listed in the
`ITEM_PIPELINES` setting, normally in your project’s
`settings.py` file (see activating-item-pipeline). Adding it to
the spider or elsewhere has no effect.
To confirm that Scrapy loaded your pipeline, look for a line like this near the
start of the crawl log:
```
[scrapy.middleware] INFO: Enabled item pipelines:
['myproject.pipelines.PricePipeline']
```
If your pipeline is missing from that list, check that its import path matches
the `ITEM_PIPELINES` entry, and that the setting is not being
overridden, for example by `custom_settings` or by a
redefinition of `ITEM_PIPELINES` in `settings.py`.
#### The item is not returned
`process_item()` must return the item (or raise
`DropItem`). A common mistake is to modify the item but
forget to return it:
```python
def process_item(self, item):
ItemAdapter(item)["price"] *= 1.15
# Bug: returns None, so the next component gets None instead of the item.
```
Return the item so that the next component, and the rest of Scrapy, can keep
processing it:
```python
def process_item(self, item):
ItemAdapter(item)["price"] *= 1.15
return item
```
Source: /topics/feed-exports.md
## Feed exports
One of the most frequently required features when implementing scrapers is
being able to store the scraped data properly and, quite often, that means
generating an “export file” with the scraped data (commonly called “export
feed”) to be consumed by other systems.
Scrapy provides this functionality out of the box with the Feed Exports, which
allows you to generate feeds with the scraped items, using multiple
serialization formats and storage backends.
This page provides detailed documentation for all feed export features. If you
are looking for a step-by-step guide, check out [Zyte’s export guides](https://docs.zyte.com/web-scraping/guides/export/index.md#exporting-scraped-data).
### Serialization formats
For serializing the scraped data, the feed exports use the Item exporters. These formats are supported out of the box:
- topics-feed-format-json
- topics-feed-format-jsonlines
- topics-feed-format-csv
- topics-feed-format-xml
But you can also extend the supported format through the
`FEED_EXPORTERS` setting.
#### JSON
- Value for the `format` key in the `FEEDS` setting: `json`
- Exporter used: `JsonItemExporter`
- See this warning if you’re using JSON with
large feeds.
#### JSON lines
- Value for the `format` key in the `FEEDS` setting: `jsonlines`
- Exporter used: `JsonLinesItemExporter`
#### CSV
- Value for the `format` key in the `FEEDS` setting: `csv`
- Exporter used: `CsvItemExporter`
- To specify columns to export, their order and their column names, use
`FEED_EXPORT_FIELDS`. Other feed exporters can also use this
option, but it is important for CSV because unlike many other export
formats CSV uses a fixed header.
#### XML
- Value for the `format` key in the `FEEDS` setting: `xml`
- Exporter used: `XmlItemExporter`
#### Pickle
- Value for the `format` key in the `FEEDS` setting: `pickle`
- Exporter used: `PickleItemExporter`
#### Marshal
- Value for the `format` key in the `FEEDS` setting: `marshal`
- Exporter used: `MarshalItemExporter`
### Storages
When using the feed exports you define where to store the feed using one or multiple [URIs](https://en.wikipedia.org/wiki/Uniform_Resource_Identifier)
(through the `FEEDS` setting). The feed exports supports multiple
storage backend types which are defined by the URI scheme.
The storages backends supported out of the box are:
- topics-feed-storage-fs
- feed-storage-ftp
- feed-storage-ftps
- topics-feed-storage-s3 (requires the s3 extra)
- topics-feed-storage-gcs (requires the gcs extra)
- topics-feed-storage-stdout
Some storage backends may be unavailable if the required extras
are not installed. For example, the S3 backend requires the s3
extra.
### Storage URI parameters
The storage URI can also contain parameters that get replaced when the feed is
being created. These parameters are:
- `%(time)s` - gets replaced by a timestamp when the feed is being created
- `%(name)s` - gets replaced by the spider name
Any other named parameter gets replaced by the spider attribute of the same
name. For example, `%(site_id)s` would get replaced by the `spider.site_id`
attribute the moment the feed is being created.
Here are some examples to illustrate:
- Store in FTP using one directory per spider:
- `ftp://user:password@ftp.example.com/scraping/feeds/%(name)s/%(time)s.json`
- Store in S3 using one directory per spider:
- `s3://mybucket/scraping/feeds/%(name)s/%(time)s.json`
> [!NOTE]
> Spider arguments become spider attributes, hence
> they can also be used as storage URI parameters.
> [!NOTE]
> Only `%(...)s` parameters are replaced. Any other percent
> character is kept as-is, so percent-encoded URIs (e.g. `%20` for a
> space or percent-encoded FTP credentials) and `pathlib.Path`
> keys containing `%(...)s` parameters both work as expected.
### Storage backends
#### Local filesystem
The feeds are stored in the local filesystem.
- URI scheme: `file`
- Example URI: `file:///tmp/export.csv`
- Required external libraries: none
Note that for the local filesystem storage (only) you can omit the scheme if
you specify a path (e.g. `/tmp/export.csv`).
Alternatively you can also use a `pathlib.Path` object.
#### FTP
The feeds are stored in a FTP server.
- URI scheme: `ftp`
- Example URI: `ftp://user:pass@ftp.example.com/path/to/export.csv`
- Required external libraries: none
FTP sends credentials and data in cleartext. Use feed-storage-ftps
instead where possible.
FTP supports two different connection modes: [active or passive](https://stackoverflow.com/a/1699163). Scrapy uses the passive connection
mode by default. To use the active connection mode instead, set the
`FEED_STORAGE_FTP_ACTIVE` setting to `True`.
The default value for the `overwrite` key in the `FEEDS` for this
storage backend is: `True`.
> [!CAUTION]
> The value `True` in `overwrite` will cause you to lose the
> previous version of your data.
This storage backend uses delayed file delivery.
#### FTPS
The feeds are stored in a FTP server, over a TLS connection, with the
certificate of the server verified.
> [!WARNING]
> Added in version 2.18.0.
- URI scheme: `ftps`
- Example URI: `ftps://user:pass@ftp.example.com/path/to/export.csv`
- Required external libraries: none
See feed-storage-ftp for connection modes, the `overwrite` default and
file delivery.
> [!NOTE]
> For SFTP, an unrelated protocol built on SSH, use
> [scrapy-feedexporter-sftp](https://github.com/scrapy-plugins/scrapy-feedexporter-sftp).
#### S3
The feeds are stored on [Amazon S3](https://aws.amazon.com/s3/).
- URI scheme: `s3`
- Example URIs:
- `s3://mybucket/path/to/export.csv`
- `s3://aws_key:aws_secret@mybucket/path/to/export.csv`
- Required extras: s3
The AWS credentials can be passed as user/password in the URI, or they can be
passed through the following settings:
- `AWS_ACCESS_KEY_ID`
- `AWS_SECRET_ACCESS_KEY`
- `AWS_SESSION_TOKEN` (only needed for [temporary security credentials](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html))
You can also define a custom ACL, custom endpoint, region name and connection
pool size for exported feeds using these settings:
- `FEED_STORAGE_S3_ACL`
- `AWS_ENDPOINT_URL`
- `AWS_REGION_NAME`
- `AWS_MAX_POOL_CONNECTIONS`
The default value for the `overwrite` key in the `FEEDS` for this
storage backend is: `True`.
> [!CAUTION]
> The value `True` in `overwrite` will cause you to lose the
> previous version of your data.
This storage backend uses delayed file delivery.
#### Google Cloud Storage (GCS)
The feeds are stored on [Google Cloud Storage](https://cloud.google.com/storage/).
- URI scheme: `gs`
- Example URIs:
- `gs://mybucket/path/to/export.csv`
- Required extras: gcs
For more information about authentication, please refer to [Google Cloud documentation](https://docs.cloud.google.com/docs/authentication).
You can set a *Project ID* and *Access Control List (ACL)* through the following settings:
- `FEED_STORAGE_GCS_ACL`
- `GCS_PROJECT_ID`
The default value for the `overwrite` key in the `FEEDS` for this
storage backend is: `True`.
> [!CAUTION]
> The value `True` in `overwrite` will cause you to lose the
> previous version of your data.
This storage backend uses delayed file delivery.
#### Standard output
The feeds are written to the standard output of the Scrapy process.
- URI scheme: `stdout`
- Example URI: `stdout:`
- Required external libraries: none
#### Delayed file delivery
As indicated above, some of the described storage backends use delayed file
delivery.
These storage backends do not upload items to the feed URI as those items are
scraped. Instead, Scrapy writes items into a temporary local file, and only
once all the file contents have been written (i.e. at the end of the crawl) is
that file uploaded to the feed URI.
If you want item delivery to start earlier when using one of these storage
backends, use `FEED_EXPORT_BATCH_ITEM_COUNT` to split the output items
in multiple files, with the specified maximum item count per file. That way, as
soon as a file reaches the maximum item count, that file is delivered to the
feed URI, allowing item delivery to start way before the end of the crawl.
### Item filtering
You can filter items that you want to allow for a particular feed by using the
`item_classes` option in feeds options. Only items of
the specified types will be added to the feed.
The `item_classes` option is implemented by the `ItemFilter`
class, which is the default value of the `item_filter` feed option.
You can create your own custom filtering class by implementing `ItemFilter`’s
method `accepts` and taking `feed_options` as an argument.
For instance:
```python
class MyCustomFilter:
def __init__(self, feed_options):
self.feed_options = feed_options
def accepts(self, item):
if "field1" in item and item["field1"] == "expected_data":
return True
return False
```
You can assign your custom filtering class to the `item_filter` option of a feed.
See `FEEDS` for examples.
#### ItemFilter
#### *class* scrapy.extensions.feedexport.ItemFilter(feed_options: dict[str, Any] | None)
This will be used by FeedExporter to decide if an item should be allowed
to be exported to a particular feed.
* **Parameters:**
**feed_options** (*dict*) – feed specific options passed from FeedExporter
##### accepts(item: Any) → bool
Return `True` if item should be exported or `False` otherwise.
* **Parameters:**
**item** (Scrapy items) – scraped item which user wants to check if is acceptable
* **Returns:**
True if accepted, False otherwise
* **Return type:**
bool
### Post-Processing
Scrapy provides an option to activate plugins to post-process feeds before they are exported
to feed storages. In addition to using builtin plugins, you
can create your own plugins.
These plugins can be activated through the `postprocessing` option of a feed.
The option must be passed a list of post-processing plugins in the order you want
the feed to be processed. These plugins can be declared either as an import string
or with the imported class of the plugin. Parameters to plugins can be passed
through the feed options. See feed options for examples.
#### Built-in Plugins
#### *class* scrapy.extensions.postprocessing.GzipPlugin(file: BinaryIO, feed_options: dict[str, Any])
Compresses received data using [gzip](https://en.wikipedia.org/wiki/Gzip).
Accepted `feed_options` parameters:
- gzip_compresslevel
- gzip_mtime
- gzip_filename
See `gzip.GzipFile` for more info about parameters.
#### *class* scrapy.extensions.postprocessing.LZMAPlugin(file: BinaryIO, feed_options: dict[str, Any])
Compresses received data using [lzma](https://en.wikipedia.org/wiki/Lempel–Ziv–Markov_chain_algorithm).
Accepted `feed_options` parameters:
- lzma_format
- lzma_check
- lzma_preset
- lzma_filters
> [!NOTE]
> `lzma_filters` cannot be used in pypy version 7.3.1 and older.
See `lzma.LZMAFile` for more info about parameters.
#### *class* scrapy.extensions.postprocessing.Bz2Plugin(file: BinaryIO, feed_options: dict[str, Any])
Compresses received data using [bz2](https://en.wikipedia.org/wiki/Bzip2).
Accepted `feed_options` parameters:
- bz2_compresslevel
See `bz2.BZ2File` for more info about parameters.
#### Custom Plugins
Each plugin is a class that must implement the following methods:
#### \_\_init_\_(self, file, feed_options)
Initialize the plugin.
* **Parameters:**
* **file** – file-like object having at least the write, tell and close methods implemented
* **feed_options** (`dict`) – feed-specific options
#### write(self, data)
Process and write data (`bytes` or `memoryview`) into the plugin’s target file.
It must return number of bytes written.
#### close(self)
Clean up the plugin.
For example, you might want to close a file wrapper that you might have
used to compress data written into the file received in the `__init__`
method.
> [!WARNING]
> Do not close the file from the `__init__` method.
To pass a parameter to your plugin, use feed options. You
can then access those parameters from the `__init__` method of your plugin.
### Settings
These are the settings used for configuring the feed exports:
- `FEEDS` (mandatory)
- `FEED_EXPORT_ENCODING`
- `FEED_STORE_EMPTY`
- `FEED_EXPORT_FIELDS`
- `FEED_EXPORT_INDENT`
- `FEED_STORAGES`
- `FEED_STORAGE_FTP_ACTIVE`
- `FEED_STORAGE_S3_ACL`
- `FEED_EXPORTERS`
- `FEED_EXPORT_BATCH_ITEM_COUNT`
#### FEEDS
Default: `{}`
A dictionary in which every key is a feed URI (or a `pathlib.Path`
object) and each value is a nested dictionary containing configuration
parameters for the specific feed.
This setting is required for enabling the feed export feature.
See topics-feed-storage-backends for supported URI schemes.
For instance:
```python
{
"items.json": {
"format": "json",
"encoding": "utf8",
"store_empty": False,
"item_classes": [MyItemClass1, "myproject.items.MyItemClass2"],
"fields": None,
"indent": 4,
"item_export_kwargs": {
"export_empty_fields": True,
},
},
"/home/user/documents/items.xml": {
"format": "xml",
"fields": ["name", "price"],
"item_filter": MyCustomFilter1,
"encoding": "latin1",
"indent": 8,
},
pathlib.Path("items.csv.gz"): {
"format": "csv",
"fields": ["price", "name"],
"item_filter": "myproject.filters.MyCustomFilter2",
"postprocessing": [MyPlugin1, "scrapy.extensions.postprocessing.GzipPlugin"],
"gzip_compresslevel": 5,
},
}
```
The following is a list of the accepted keys and the setting that is used
as a fallback value if that key is not provided for a specific feed definition:
- `format`: the serialization format.
If not set, it is inferred from the file extension of the feed URI, e.g.
`json` for a URI ending in `.json`. It is mandatory if it cannot
be inferred this way.
- `batch_item_count`: falls back to
`FEED_EXPORT_BATCH_ITEM_COUNT`.
- `encoding`: falls back to `FEED_EXPORT_ENCODING`.
- `fields`: falls back to `FEED_EXPORT_FIELDS`.
- `item_classes`: list of item classes to export.
If undefined or empty, all items are exported.
- `item_filter`: a filter class to filter items to export.
`ItemFilter` is used be default.
- `indent`: falls back to `FEED_EXPORT_INDENT`.
- `item_export_kwargs`: `dict` with keyword arguments for the corresponding item exporter class.
- `overwrite`: whether to overwrite the file if it already exists
(`True`) or append to its content (`False`).
The default value depends on the storage backend:
- topics-feed-storage-fs: `False`
- feed-storage-ftp and feed-storage-ftps: `True`
> [!NOTE]
> Some FTP servers may not support appending to files (the
> `APPE` FTP command).
- topics-feed-storage-s3: `True` (appending is not supported)
- topics-feed-storage-gcs: `True` (appending is not supported)
- topics-feed-storage-stdout: `False` (overwriting is not supported)
- `store_empty`: falls back to `FEED_STORE_EMPTY`.
- `uri_params`: falls back to `FEED_URI_PARAMS`.
- `postprocessing`: list of plugins to use for post-processing.
The plugins will be used in the order of the list passed.
#### FEED_EXPORT_ENCODING
Default: `"utf-8"` (fallback: `None`)
The encoding to be used for the feed.
If set to `None`, it uses UTF-8 for everything except JSON output, which uses
safe numeric encoding (`\uXXXX` sequences) for historic reasons.
Use `"utf-8"` if you want UTF-8 for JSON too.
#### FEED_EXPORT_FIELDS
Default: `None`
Use the `FEED_EXPORT_FIELDS` setting to define the fields to export, their
order and their output names. See `BaseItemExporter.fields_to_export` for more information.
#### FEED_EXPORT_INDENT
Default: `0`
Amount of spaces used to indent the output on each level. If `FEED_EXPORT_INDENT`
is a non-negative integer, then array elements and object members will be pretty-printed
with that indent level. An indent level of `0` (the default), or negative,
will put each item on a new line. `None` selects the most compact representation.
Currently implemented only by `JsonItemExporter`
and `XmlItemExporter`, i.e. when you are exporting
to `.json` or `.xml`.
#### FEED_STORE_EMPTY
Default: `True`
Whether to export empty feeds (i.e. feeds with no items).
If `False`, and there are no items to export, no new files are created and
existing files are not modified, even if the overwrite feed option is enabled.
#### FEED_STORAGES
Default: `{}`
A dict containing additional feed storage backends supported by your project.
The keys are URI schemes and the values are paths to storage classes.
#### FEED_STORAGE_FTP_ACTIVE
Default: `False`
Whether to use the active connection mode when exporting feeds to an FTP server
(`True`) or use the passive connection mode instead (`False`, default).
For information about FTP connection modes, see [What is the difference between
active and passive FTP?](https://stackoverflow.com/a/1699163).
#### FEED_STORAGE_S3_ACL
Default: `''` (empty string)
A string containing a custom ACL for feeds exported to Amazon S3 by your project.
For a complete list of available values, access the [Canned ACL](https://docs.aws.amazon.com/AmazonS3/latest/userguide/acl-overview.html#canned-acl) section on Amazon S3 docs.
#### FEED_STORAGES_BASE
Default:
```python
{
"": "scrapy.extensions.feedexport.FileFeedStorage",
"file": "scrapy.extensions.feedexport.FileFeedStorage",
"stdout": "scrapy.extensions.feedexport.StdoutFeedStorage",
"s3": "scrapy.extensions.feedexport.S3FeedStorage",
"gs": "scrapy.extensions.feedexport.GCSFeedStorage",
"ftp": "scrapy.extensions.feedexport.FTPFeedStorage",
"ftps": "scrapy.extensions.feedexport.FTPFeedStorage",
}
```
A dict containing the built-in feed storage backends supported by Scrapy. You
can disable any of these backends by assigning `None` to their URI scheme in
`FEED_STORAGES`. E.g., to disable the built-in FTP storage backend
(without replacement), place this in your `settings.py`:
```python
FEED_STORAGES = {
"ftp": None,
}
```
#### FEED_EXPORTERS
Default: `{}`
A dict containing additional exporters supported by your project. The keys are
serialization formats and the values are paths to Item exporter classes.
#### FEED_EXPORTERS_BASE
Default:
```python
{
"json": "scrapy.exporters.JsonItemExporter",
"jsonlines": "scrapy.exporters.JsonLinesItemExporter",
"jsonl": "scrapy.exporters.JsonLinesItemExporter",
"jl": "scrapy.exporters.JsonLinesItemExporter",
"csv": "scrapy.exporters.CsvItemExporter",
"xml": "scrapy.exporters.XmlItemExporter",
"marshal": "scrapy.exporters.MarshalItemExporter",
"pickle": "scrapy.exporters.PickleItemExporter",
}
```
A dict containing the built-in feed exporters supported by Scrapy. You can
disable any of these exporters by assigning `None` to their serialization
format in `FEED_EXPORTERS`. E.g., to disable the built-in CSV exporter
(without replacement), place this in your `settings.py`:
```python
FEED_EXPORTERS = {
"csv": None,
}
```
#### FEED_EXPORT_BATCH_ITEM_COUNT
Default: `0`
If assigned an integer number higher than `0`, Scrapy generates multiple output files
storing up to the specified number of items in each output file.
When generating multiple output files, you must use at least one of the following
placeholders in the feed URI to indicate how the different output file names are
generated:
* `%(batch_time)s` - gets replaced by a timestamp when the feed is being created
(e.g. `2020-03-28T14-45-08.237134`)
* `%(batch_id)d` - gets replaced by the 1-based sequence number of the batch.
Use printf-style string formatting to
alter the number format. For example, to make the batch ID a 5-digit
number by introducing leading zeroes as needed, use `%(batch_id)05d`
(e.g. `3` becomes `00003`, `123` becomes `00123`).
For instance, if your settings include:
```python
FEED_EXPORT_BATCH_ITEM_COUNT = 100
```
And your `crawl` command line is:
```
scrapy crawl spidername -o "dirname/%(batch_id)d-filename%(batch_time)s.json"
```
The command line above can generate a directory tree like:
```
->projectname
-->dirname
--->1-filename2020-03-28T14-45-08.237134.json
--->2-filename2020-03-28T14-45-09.148903.json
--->3-filename2020-03-28T14-45-10.046092.json
```
Where the first and second files contain exactly 100 items. The last one contains
100 items or fewer.
#### FEED_URI_PARAMS
Default: `None`
A string with the import path of a function to set the parameters to apply with
printf-style string formatting to the
feed URI.
The function signature should be as follows:
#### scrapy.extensions.feedexport.uri_params(params, spider)
Return a `dict` of key-value pairs to apply to the feed URI using
printf-style string formatting.
* **Parameters:**
* **params** (*dict*) –
default key-value pairs
Specifically:
- `batch_id`: ID of the file batch. See
`FEED_EXPORT_BATCH_ITEM_COUNT`.
If `FEED_EXPORT_BATCH_ITEM_COUNT` is `0`, `batch_id`
is always `1`.
- `batch_time`: UTC date and time, in ISO format with `:`
replaced with `-`.
See `FEED_EXPORT_BATCH_ITEM_COUNT`.
- `time`: `batch_time`, with microseconds set to `0`.
* **spider** (*scrapy.Spider*) – source spider of the feed items
> [!CAUTION]
> The function must return a new dictionary instead of modifying
> the received `params` in-place.
For example, to include the `name` of the
source spider in the feed URI:
1. Define the following function somewhere in your project:
```python
# myproject/utils.py
def uri_params(params, spider):
return {**params, "spider_name": spider.name}
```
2. Point `FEED_URI_PARAMS` to that function in your settings:
```python
# myproject/settings.py
FEED_URI_PARAMS = "myproject.utils.uri_params"
```
3. Use `%(spider_name)s` in your feed URI:
```
scrapy crawl <spider_name> -o "%(spider_name)s.jsonl"
```
Source: /topics/request-response.md
## Requests and Responses
Scrapy uses `Request` and `Response` objects for crawling web
sites.
Typically, `Request` objects are generated in the spiders and pass
across the system until they reach the Downloader, which executes the request
and returns a `Response` object which travels back to the spider that
issued the request.
Both `Request` and `Response` classes have subclasses which add
functionality not required in the base classes. These are described
below in topics-request-response-ref-request-subclasses and
topics-request-response-ref-response-subclasses.
### Request objects
#### *class* scrapy.Request(\*args: Any, \*\*kwargs: Any)
Represents an HTTP request, which is usually generated in a Spider and
executed by the Downloader, thus generating a `Response`.
* **Parameters:**
* **url** (*str*) –
the URL of this request
If the URL is invalid, a `ValueError` exception is raised.
* **callback** (*Callable* *[**Concatenate* *[**Response* *,* *...* *]* *,* *Any* *]* *|* *None*) – sets `callback`, defaults to `None`.
* **method** (*str*) – the HTTP method of this request. Defaults to `'GET'`.
* **meta** (*dict*) – the initial values for the `Request.meta` attribute. If
given, the dict passed in this parameter will be shallow copied.
* **body** (*bytes* *or* *str*) – the request body. If a string is passed, then it’s encoded as
bytes using the `encoding` passed (which defaults to `utf-8`). If
`body` is not given, an empty bytes object is stored. Regardless of the
type of this argument, the final value stored will be a bytes object
(never a string or `None`).
* **headers** (*dict*) –
the headers of this request. The dict values can be strings
(for single valued headers) or lists (for multi-valued headers). If
`None` is passed as value, the HTTP header will not be sent at all.
> [!CAUTION]
> Cookies set via the `Cookie` header are not considered by the
> cookie middleware. If you need to set cookies for a
> request, use the `cookies` argument.
* **cookies** (*dict* *or* *list*) – the request cookies, as a dict of cookie names and values
or as a list of dicts with a cookie each. See cookies.
* **encoding** (*str*) –
the encoding of this request (defaults to `'utf-8'`).
This encoding will be used to percent-encode the URL and to convert the
body to bytes (if given as a string).
To disable URL percent-encoding for a request, use the
`verbatim_url` request meta key.
* **priority** (*int*) – sets `priority`, defaults to `0`.
* **dont_filter** (*bool*) – sets `dont_filter`, defaults to `False`.
* **errback** (*Callable* *[* *[**Failure* *]* *,* *Any* *]* *|* *None*) – sets `errback`, defaults to `None`.
* **flags** (*list*) – Flags sent to the request, can be used for logging or similar purposes.
* **cb_kwargs** (*dict*) – A dict with arbitrary data that will be passed as keyword arguments to the Request’s callback.
##### url
A string containing the URL of this request.
Keep in mind that this attribute contains the escaped URL, so it can
differ from the URL passed in the `__init__()` method.
If `verbatim_url` is set to `True`, the URL is kept as
passed to `__init__()`.
This attribute is read-only. To change the URL of a Request use
`replace()`.
##### method
A string representing the HTTP method in the request. This is guaranteed to
be uppercase. Example: `"GET"`, `"POST"`, `"PUT"`, etc
##### headers
A dictionary-like (`scrapy.http.headers.Headers`) object which contains
the request headers.
##### body
The request body as bytes.
This attribute is read-only. To change the body of a Request use
`replace()`.
##### callback *: CallbackT | None*
`Callable` to parse the
`Response` to this request once received.
The callable must expect the response as its first parameter, and
support any additional keyword arguments set through
`cb_kwargs`. See writing-callbacks and
callback-output.
In addition to an arbitrary callable, the following values are also
supported:
- `None` (default), which indicates that the
`parse()` method of the spider must be used.
- `NO_CALLBACK()`.
If an unhandled exception is raised during request or response
processing, i.e. by a spider middleware, downloader middleware or download handler
(`DOWNLOAD_HANDLERS`), `errback` is called instead.
> [!TIP]
> `HttpErrorMiddleware`
> raises exceptions for non-2xx responses by default, sending them
> to the `errback` instead.
> [!NOTE]
> ### See also
>
> callbacks
##### errback *: Callable[[Failure], Any] | None*
`Callable` to handle exceptions raised
during request or response processing.
The callable must expect a `Failure` as
its first parameter.
> [!NOTE]
> ### See also
>
> errbacks
##### priority *: int*
Default: `0`
Value that the scheduler may use for
request prioritization.
Built-in schedulers prioritize requests with a higher priority
value.
Negative values are allowed.
##### cb_kwargs
A dictionary that contains arbitrary metadata for this request. Its contents
will be passed to the Request’s callback as keyword arguments. It is empty
for new Requests, which means by default callbacks only get a
`Response` object as argument.
This dict is shallow copied when the request is
cloned using the `copy()` or `replace()` methods, and can also be
accessed, in your spider, from the `response.cb_kwargs` attribute.
In case of a failure to process the request, this dict can be accessed as
`failure.request.cb_kwargs` in the request’s errback. For more information,
see errback-cb_kwargs.
> [!NOTE]
> When `JOBDIR` is set, requests are serialized to disk
> with `pickle` (see request-serialization). As a result,
> the callback receives a deep copy of any object stored in
> `cb_kwargs`, so mutating such an object in the callback does not
> affect the original. Avoid relying on shared mutable state passed
> through `cb_kwargs` in that case.
##### meta *= {}*
> A dictionary of arbitrary metadata for the request.
> You may extend request metadata as you see fit.
> Request metadata can also be accessed through the
> `meta` attribute of a response.
> To pass your own data from one spider callback to another, use
> `cb_kwargs` instead, see callback-data. However, request
> metadata may be the right choice in certain scenarios, such as to
> maintain some debugging data across all follow-up requests (e.g. the
> source URL).
> A common use of request metadata is to define request-specific
> parameters for Scrapy components (extensions, middlewares, etc.). For
> example, if you set `dont_retry` to `True`,
> `RetryMiddleware` will never
> retry that request, even if it fails. See topics-request-meta.
> You may also use request metadata in your custom Scrapy components, for
> example, to keep request state information relevant to your component.
> For example,
> `RetryMiddleware` uses the
> `retry_times` metadata key to keep track of how many times a
> request has been retried so far.
> Copying all the metadata of a previous request into a new, follow-up
> request in a spider callback is a bad practice, because request
> metadata may include metadata set by Scrapy components that is not
> meant to be copied into other requests. For example, copying the
> `retry_times` metadata key into follow-up requests can lower
> the amount of retries allowed for those follow-up requests.
> You should only copy all request metadata from one request to another
> if the new request is meant to replace the old request, as is often the
> case when returning a request from a downloader middleware method.
> Also mind that the `copy()` and `replace()` request methods
> shallow-copy request metadata.
> > [!NOTE]
> > ### See also
> >
> > `MetaCopyDetectionMiddleware`
> > for a built-in middleware that warns about this issue at run time.
##### dont_filter *: bool*
Whether this request may be filtered out by components that support filtering out requests (`False`,
default), or those components should not filter out this request
(`True`).
The following built-in components check this attribute:
- The scheduler uses it to skip
duplicate request filtering (see
`DUPEFILTER_CLASS`). When set to `True`, the
request is not checked against the duplicate filter,
allowing requests that would otherwise be considered duplicates
to be scheduled multiple times.
- `OffsiteMiddleware`
uses it to allow requests to domains not in
`allowed_domains`. To skip only the offsite
filter without affecting other components, consider using the
`allow_offsite` request meta key instead.
Third-party components may also use this attribute to decide whether
to filter out a request.
When defining the start URLs of a spider through
`start_urls`, this attribute is enabled by
default. See `start()`.
##### attributes *: tuple[str, ...]* *= ('url', 'headers', 'body', 'cookies', 'meta', 'encoding', 'flags', 'cb_kwargs', 'callback', 'dont_filter', 'errback', 'method', 'priority')*
A tuple of `str` objects containing the name of all public
attributes of the class that are also keyword parameters of the
`__init__()` method.
Currently used by `Request.replace()`, `Request.to_dict()` and
`request_from_dict()`.
##### copy()
Return a new Request which is a copy of this Request. See also:
callback-data.
##### replace([urlmethodheadersbodycookiesmetaflagsencodingprioritydont_filtercallbackerrbackcb_kwargscls], , , , , , , , , , , , ,)
Return a Request object with the same members, except for those members
given new values by whichever keyword arguments are specified. The
`cb_kwargs` and `meta` attributes are shallow
copied by default (unless new values are given as arguments). See also
callback-data.
##### *classmethod* from_curl(curl_command: str, ignore_unknown_options: bool = True, \*\*kwargs: Any) → Self
Create a Request object from a string containing a [cURL](https://curl.se/) command. It populates the HTTP method, the
URL, the headers, the cookies and the body. It accepts the same
arguments as the `Request` class, taking preference and
overriding the values of the same arguments contained in the cURL
command.
Unrecognized options are ignored by default. To raise an error when
finding unknown options call this method by passing
`ignore_unknown_options=False`.
> [!CAUTION]
> Using `from_curl()` from `Request`
> subclasses, such as `JsonRequest`, or
> `XmlRpcRequest`, as well as having
> downloader middlewares
> and
> spider middlewares
> enabled, such as
> `DefaultHeadersMiddleware`,
> `UserAgentMiddleware`,
> or
> `HttpCompressionMiddleware`,
> may modify the `Request` object.
To translate a cURL command into a Scrapy request,
you may use [curl2scrapy](https://michael-shub.github.io/curl2scrapy/).
##### to_curl() → str
Return a string with a [cURL](https://curl.se/) command equivalent
to this request.
Inverse of `from_curl()`. See also
`scrapy.utils.request.request_to_curl()`.
> [!WARNING]
> Added in version 2.18.0.
##### to_dict(\*, spider: Spider | None = None) → dict[str, Any]
Return a dictionary containing the Request’s data.
Use `request_from_dict()` to convert back into a `Request` object.
If a spider is given, this method will try to find out the name of the spider methods used as callback
and errback and include them in the output dict, raising an exception if they cannot be found.
#### Creating requests that submit HTML forms
Use form2request to build request data from an HTML
`<form>` element and convert it to a `Request`.
Install it with pip:
```bash
pip install form2request
```
Select the desired form with CSS or XPath, then build and convert request
data:
```python
from form2request import form2request
def parse(self, response):
form = response.css("form#search")
request_data = form2request(form, data={"q": "scrapy"})
yield request_data.to_scrapy(callback=self.parse_results)
```
Use `data` to override field values. To drop a field from the resulting
request, set its value to `None`.
By default, form2request simulates clicking the first submit button. To submit
without clicking any button, pass `click=False`. To click a specific submit
button, pass its element:
```python
def parse(self, response):
form = response.css("form#checkout")
submit = form.css('button[name="pay"]')
request_data = form2request(form, click=submit)
```
##### Using form2request to simulate a user login
It is usual for web sites to provide pre-populated form fields through `<input
type="hidden">` elements, such as session related data or authentication
tokens (for login pages). Build the request from the form and only override the
credentials:
```python
import scrapy
from form2request import form2request
class LoginSpider(scrapy.Spider):
name = "example.com"
start_urls = ["http://www.example.com/users/login.php"]
def parse(self, response):
form = response.css("form")
request_data = form2request(
form,
data={"username": "john", "password": "secret"},
)
yield request_data.to_scrapy(callback=self.after_login)
def after_login(self, response): ...
```
#### Other functions related to requests
#### scrapy.http.request.NO_CALLBACK(\*args: Any, \*\*kwargs: Any) → NoReturn
When assigned to the `callback` parameter of
`Request`, it indicates that the request is not meant
to have a spider callback at all.
For example:
```python
Request("https://example.com", callback=NO_CALLBACK)
```
This value should be used by components that
create and handle their own requests, e.g. through
`scrapy.core.engine.ExecutionEngine.download()`, so that downloader
middlewares handling such requests can treat them differently from requests
intended for the `parse()` callback.
#### scrapy.utils.request.request_from_dict(d: dict[str, Any], \*, spider: Spider | None = None) → Request
Create a `Request` object from a dict.
If a spider is given, it will try to resolve the callbacks looking at the
spider for methods with the same name.
#### scrapy.utils.httpobj.urlparse_cached(request_or_response: Request | Response) → ParseResult
Return the result of parsing the URL of *request_or_response*, a
`Request` or `Response` object, with
`urllib.parse.urlparse()`.
The result is cached, using a `weakref.WeakKeyDictionary` keyed on
*request_or_response*, so that the URL of a given object is parsed only
once. Prefer this function over calling `urllib.parse.urlparse()` on
`request_or_response.url` directly when the same URL may be parsed more
than once.
#### Request fingerprints
There are some aspects of scraping, such as filtering out duplicate requests
(see `DUPEFILTER_CLASS`) or caching responses (see
`HTTPCACHE_POLICY`), where you need the ability to generate a short,
unique identifier from a `Request` object: a request
fingerprint.
You often do not need to worry about request fingerprints, the default request
fingerprinter works for most projects.
However, there is no universal way to generate a unique identifier from a
request, because different situations require comparing requests differently.
For example, sometimes you may need to compare URLs case-insensitively, include
URL fragments, exclude certain URL query parameters, include some or all
headers, etc.
To change how request fingerprints are built for your requests, use the
`REQUEST_FINGERPRINTER_CLASS` setting.
##### REQUEST_FINGERPRINTER_CLASS
Default: `scrapy.utils.request.RequestFingerprinter`
A request fingerprinter class or its
import path.
#### *class* scrapy.utils.request.RequestFingerprinter(crawler: Crawler | None = None)
Default fingerprinter.
It takes into account a canonical version
(`w3lib.url.canonicalize_url()`) of `request.url` and the values of `request.method` and `request.body`, unless `verbatim_url` is true for that
request. It then generates an [SHA1](https://en.wikipedia.org/wiki/SHA-1)
hash.
##### Writing your own request fingerprinter
A request fingerprinter is a component that must
implement the following method:
#### fingerprint(self, request: scrapy.Request)
Return a `bytes` object that uniquely identifies *request*.
See also request-fingerprint-restrictions.
The `fingerprint()` method of the default request fingerprinter,
`scrapy.utils.request.RequestFingerprinter`, uses
`scrapy.utils.request.fingerprint()` with its default parameters. For some
common use cases you can use `scrapy.utils.request.fingerprint()` as well
in your `fingerprint()` method implementation:
#### scrapy.utils.request.fingerprint(request: Request, \*, include_headers: Iterable[bytes | str] | None = None, keep_fragments: bool = False) → bytes
Return the request fingerprint.
The request fingerprint is a hash that uniquely identifies the resource the
request points to. For example, take the following two urls:
`http://www.example.com/query?id=111&cat=222`,
`http://www.example.com/query?cat=222&id=111`.
Even though those are two different URLs both point to the same resource
and are equivalent (i.e. they should return the same response).
Another example are cookies used to store session ids. Suppose the
following page is only accessible to authenticated users:
`http://www.example.com/members/offers.html`.
Lots of sites use a cookie to store the session id, which adds a random
component to the HTTP Request and thus should be ignored when calculating
the fingerprint.
For this reason, request headers are ignored by default when calculating
the fingerprint. If you want to include specific headers use the
include_headers argument, which is a list of Request headers to include.
Also, servers usually ignore fragments in urls when handling requests,
so they are also ignored by default when calculating the fingerprint.
If you want to include them, set the keep_fragments argument to True
(for instance when handling requests with a headless browser).
keep_fragments is not a substitute for rendering JavaScript to reach
content loaded based on the URL fragment; see
topics-dynamic-content for that instead.
By default, request fingerprinting canonicalizes the request URL. If
`verbatim_url` is set to `True`, fingerprinting does not
canonicalize the URL, and the `keep_fragments` parameter is ignored (it is
effectively true).
For example, to take the value of a request header named `X-ID` into
account:
```python
# my_project/settings.py
REQUEST_FINGERPRINTER_CLASS = "my_project.utils.RequestFingerprinter"
# my_project/utils.py
from scrapy.utils.request import fingerprint
class RequestFingerprinter:
def fingerprint(self, request):
return fingerprint(request, include_headers=["X-ID"])
```
You can also write your own fingerprinting logic from scratch.
However, if you do not use `scrapy.utils.request.fingerprint()`, make sure
you use `WeakKeyDictionary` to cache request fingerprints:
- Caching saves CPU by ensuring that fingerprints are calculated only once
per request, and not once per Scrapy component that needs the fingerprint
of a request.
- Using `WeakKeyDictionary` saves memory by ensuring that
request objects do not stay in memory forever just because you have
references to them in your cache dictionary.
For example, to take into account only the URL of a request, without any prior
URL canonicalization or taking the request method or body into account:
```python
from hashlib import sha1
from weakref import WeakKeyDictionary
from scrapy.utils.python import to_bytes
class RequestFingerprinter:
cache = WeakKeyDictionary()
def fingerprint(self, request):
if request not in self.cache:
fp = sha1()
fp.update(to_bytes(request.url))
self.cache[request] = fp.digest()
return self.cache[request]
```
If you need to be able to override the request fingerprinting for arbitrary
requests from your spider callbacks, you may implement a request fingerprinter
that reads fingerprints from `request.meta`
when available, and then falls back to
`scrapy.utils.request.fingerprint()`. For example:
```python
from scrapy.utils.request import fingerprint
class RequestFingerprinter:
def fingerprint(self, request):
if "fingerprint" in request.meta:
return request.meta["fingerprint"]
return fingerprint(request)
```
If you need to reproduce the same fingerprinting algorithm as Scrapy 2.6, use
the following request fingerprinter:
```python
from hashlib import sha1
from weakref import WeakKeyDictionary
from scrapy.utils.python import to_bytes
from w3lib.url import canonicalize_url
class RequestFingerprinter:
cache = WeakKeyDictionary()
def fingerprint(self, request):
if request not in self.cache:
fp = sha1()
fp.update(to_bytes(request.method))
fp.update(to_bytes(canonicalize_url(request.url)))
fp.update(request.body or b"")
self.cache[request] = fp.digest()
return self.cache[request]
```
##### Request fingerprint restrictions
Scrapy components that use request fingerprints may impose additional
restrictions on the format of the fingerprints that your request
fingerprinter generates.
The following built-in Scrapy components have such restrictions:
- `scrapy.extensions.httpcache.FilesystemCacheStorage` (default
value of `HTTPCACHE_STORAGE`)
Request fingerprints must be at least 1 byte long.
Path and filename length limits of the file system of
`HTTPCACHE_DIR` also apply. Inside `HTTPCACHE_DIR`,
the following directory structure is created:
- `Spider.name`
- first byte of a request fingerprint as hexadecimal
- fingerprint as hexadecimal
- filenames up to 16 characters long
For example, if a request fingerprint is made of 20 bytes (default),
`HTTPCACHE_DIR` is `'/home/user/project/.scrapy/httpcache'`,
and the name of your spider is `'my_spider'` your file system must
support a file path like:
```
/home/user/project/.scrapy/httpcache/my_spider/01/0123456789abcdef0123456789abcdef01234567/response_headers
```
- `scrapy.extensions.httpcache.DbmCacheStorage`
The underlying DBM implementation must support keys as long as twice
the number of bytes of a request fingerprint, plus 5. For example,
if a request fingerprint is made of 20 bytes (default),
45-character-long keys must be supported.
### Callbacks
A callback is a function that Scrapy calls with the `Response` of a
`Request` once that request has been downloaded, so that you can
extract data from that response and generate additional requests to continue
the crawl:
```python
from scrapy import Request, Spider
class BookSpider(Spider):
name = "books"
async def start(self):
yield Request("https://books.toscrape.com/", callback=self.parse_home)
def parse_home(self, response):
for url in response.css("h3 a::attr(href)").getall():
yield Request(response.urljoin(url), callback=self.parse_book)
def parse_book(self, response):
yield {"title": response.css("h1::text").get()}
```
Requests may also define an errback, which Scrapy calls
instead of the callback when an exception is raised while processing the
request or its response, e.g. a connection error or, by default, a non-2xx
response.
#### Assigning a callback to a request
To assign a callback to a request, use the `callback` parameter of
`Request`, which sets the `Request.callback` attribute:
```python
from scrapy import Request
def parse_home(response): ...
request = Request("https://books.toscrape.com/", callback=parse_home)
```
Requests with no callback, i.e. with `callback` set to
`None`, are handled by the `parse()` method of the spider:
```python
request = Request("https://books.toscrape.com/") # Handled by parse()
```
If a request is never meant to reach a spider callback, e.g. because a
component sends it and handles its response itself,
assign the special `NO_CALLBACK()` value to it
instead, so that downloader middlewares
can tell such requests apart.
While `callback` only accepts callables, some spider
classes let you also define a callback by name: both `CrawlSpider.rules` and `SitemapSpider.sitemap_rules` accept the name of a spider
method as a string.
#### Writing a callback
Any callable can be a callback, as long as it takes the response as its first
positional parameter, and any additional callback data
as keyword parameters. Spider methods are the most common choice, but plain
functions, lambda expressions and other callable objects work as well.
> [!NOTE]
> If you enable job persistence through the
> `JOBDIR` setting, callbacks must be methods of the running spider.
> Requests with any other callback cannot be serialized, so they are kept in
> memory only and lost when you pause the crawl. See
> request-serialization.
A callback can be:
- A regular function:
```python
def parse(self, response):
return {"url": response.url}
```
- A generator function:
```python
def parse(self, response):
yield {"url": response.url}
```
- A coroutine function, i.e. defined with `async def`:
```python
async def parse(self, response):
return {"url": response.url}
```
- An asynchronous generator function:
```python
async def parse(self, response):
yield {"url": response.url}
```
The last two allow using `await`, `async for` and `async with` in your
callback. See topics-coroutines.
#### Callback output
A callback may return or yield any of the following:
- `None`, which does nothing.
Callbacks that produce no output at all, e.g. callbacks that only log
information about the response, are perfectly valid. `None` values within
an iterable of callback output are ignored as well.
- A `Request` object, which Scrapy schedules, downloads and
eventually sends to its own callback.
- An item object, which Scrapy sends to the
item pipelines.
Any object that is neither `None` nor a `Request` object
is treated as an item.
- An iterable of any of the values above, e.g. a list or, more commonly, a
generator.
Asynchronous iterables, e.g. an
asynchronous generator, are also supported.
> [!NOTE]
> When a callback *returns* an object, Scrapy iterates that object if
> it supports iteration, except for `dict`, `Item`,
> `str` and `bytes` objects, which are always handled as single
> items.
> [!NOTE]
> In a generator callback, a `return` statement with a value does not
> produce any output, since such a value is not part of what the generator
> yields. Scrapy logs a warning when it detects such a callback, see
> `WARN_ON_GENERATOR_RETURN_VALUE`.
Before Scrapy acts on the output of a callback, that output goes through the
`process_spider_output()` method
of your spider middlewares, which may modify
it or drop part of it.
If a callback raises an exception, the `errback` of the
request is *not* called. The exception goes through the
`process_spider_exception()`
method of your spider middlewares instead and, unless one of them handles it,
Scrapy logs it and sends the `spider_error` signal.
#### Passing additional data to callback functions
In some cases you may be interested in passing data to a callback in addition
to the response, e.g. data extracted from the response that triggered the
request. The following example shows how to achieve this by using the
`Request.cb_kwargs` attribute:
```python
from scrapy import Request
def parse(self, response):
request = Request(
"http://www.example.com/index.html",
callback=self.parse_page2,
cb_kwargs=dict(main_url=response.url),
)
request.cb_kwargs["foo"] = "bar" # add more arguments for the callback
yield request
def parse_page2(self, response, main_url, foo):
yield dict(
main_url=main_url,
other_url=response.url,
foo=foo,
)
```
`Request.cb_kwargs` is the recommended way to pass your own data to a
callback. Use `Request.meta` only for data aimed at components, such as middlewares and extensions.
### Errbacks
The errback of a request is a function that will be called when an exception
is raised while processing it.
It receives a `Failure` as first parameter and can
be used to track connection establishment timeouts, DNS errors etc.
Scrapy sets the `request` attribute of that
`Failure` object to the `Request`
object being processed.
If an errback raises an exception, Scrapy logs it and sends the
`spider_error` signal, unless the exception is the one that the errback
received, which Scrapy logs as a download error instead.
Here’s an example spider logging all errors and catching some specific
errors if needed:
```python
from scrapy import Request, Spider
from scrapy.spidermiddlewares.httperror import HttpError
from twisted.internet.error import DNSLookupError
from twisted.internet.error import TimeoutError, TCPTimedOutError
class ErrbackSpider(Spider):
name = "errback_example"
start_urls = [
"http://www.httpbin.org/", # HTTP 200 expected
"http://www.httpbin.org/status/404", # Not found error
"http://www.httpbin.org/status/500", # server issue
"http://www.httpbin.org:12345/", # non-responding host, timeout expected
"https://example.invalid/", # DNS error expected
]
async def start(self):
for u in self.start_urls:
yield Request(
u,
callback=self.parse_httpbin,
errback=self.errback_httpbin,
dont_filter=True,
)
def parse_httpbin(self, response):
self.logger.info(f"Got successful response from {response.url}")
# do something useful here...
def errback_httpbin(self, failure):
# log all failures
self.logger.error(repr(failure))
# in case you want to do something special for some errors,
# you may need the failure's type:
if failure.check(HttpError):
# these exceptions come from HttpError spider middleware
# you can get the non-200 response
response = failure.value.response
self.logger.error("HttpError on %s", response.url)
elif failure.check(DNSLookupError):
# this is the original request
request = failure.request
self.logger.error("DNSLookupError on %s", request.url)
elif failure.check(TimeoutError, TCPTimedOutError):
request = failure.request
self.logger.error("TimeoutError on %s", request.url)
```
#### Accessing additional data in errback functions
In case of a failure to process the request, you may be interested in
accessing arguments to the callback functions so you can process further
based on the arguments in the errback. The following example shows how to
achieve this by using `Failure.request.cb_kwargs`:
```python
from scrapy import Request
def parse(self, response):
request = Request(
"http://www.example.com/index.html",
callback=self.parse_page2,
errback=self.errback_page2,
cb_kwargs=dict(main_url=response.url),
)
yield request
def parse_page2(self, response, main_url):
pass
def errback_page2(self, failure):
yield dict(
main_url=failure.request.cb_kwargs["main_url"],
)
```
### Request.meta special keys
The `Request.meta` attribute can contain any arbitrary data, but there
are some special keys recognized by Scrapy and its built-in extensions.
Those are:
* `allow_offsite`
* `autothrottle_dont_adjust_delay`
* `bindaddress`
* `cache_timestamp`
* `cookiejar`
* `depth`
* `dont_cache`
* `dont_merge_cookies`
* `dont_obey_robotstxt`
* `dont_redirect`
* `dont_retry`
* `download_fail_on_dataloss`
* `download_latency`
* `download_maxsize`
* `download_slot`
* `download_timeout`
* `download_warnsize`
* `ftp_local_filename`
* `ftp_passive`
* `ftp_password`
* `ftp_user`
* `give_up_log_level`
* `handle_httpstatus_all`
* `handle_httpstatus_list`
* `http_auth_domain`
* `http_pass`
* `http_user`
* `is_start_request`
* `link_text`
* `max_retry_times`
* `priority_adjust`
* `proxy`
* `redirect_reasons`
* `redirect_times`
* `redirect_ttl`
* `redirect_urls`
* `referrer_policy`
* `retry_times`
* `rule`
* `verbatim_url`
Scrapy components also use meta keys whose name starts with an underscore, such
as `_auth_proxy`. Those are internal, and may change or disappear at any time.
#### bindaddress
The default local outgoing address for download-handler connections.
This meta value can be either:
- a host address as a string (e.g. `"127.0.0.2"`), in which case the local
port is chosen automatically, or
- a `(host, port)` tuple (e.g. `("127.0.0.2", 50000)`) to bind to both a
specific local interface and a specific local port.
For example:
```python
Request(
"https://example.org",
meta={"bindaddress": "127.0.0.2"},
)
```
```python
Request(
"https://example.org",
meta={"bindaddress": ("127.0.0.2", 50000)},
)
```
If not set, built-in HTTP download handlers use the value of
`DOWNLOAD_BIND_ADDRESS` as the default bind address.
Set the `bindaddress` request meta key to override it for a
specific request.
This meta key is not supported by
`HttpxDownloadHandler` and
`AiohttpDownloadHandler`, but
the `DOWNLOAD_BIND_ADDRESS` setting is supported by them.
#### download_timeout
The amount of time (in secs) that the downloader will wait before timing out.
See also: `DOWNLOAD_TIMEOUT`.
#### download_latency
The amount of time spent to fetch the response, since the request has been
started, i.e. HTTP message sent over the network. This meta key only becomes
available when the response has been downloaded. While most other meta keys are
used to control Scrapy behavior, this one is supposed to be read-only.
#### download_fail_on_dataloss
Whether or not to fail on broken responses. See:
`DOWNLOAD_FAIL_ON_DATALOSS`.
#### ftp_local_filename
Path, as `bytes`, of the file where to write the response body of an
`ftp://` request. If set, `Response.body`
holds this path instead of the file contents.
#### give_up_log_level
> [!WARNING]
> Added in version 2.17.0.
Logging level used for the message logged when a request
exceeds its retries. See `RETRY_GIVE_UP_LOG_LEVEL` for details.
#### http_auth_domain
> [!WARNING]
> Added in version 2.17.0.
Overrides `HTTPAUTH_DOMAIN` for this request.
#### http_pass
> [!WARNING]
> Added in version 2.17.0.
Overrides `HTTPAUTH_PASS` for this request.
#### http_user
> [!WARNING]
> Added in version 2.17.0.
Overrides `HTTPAUTH_USER` for this request.
#### max_retry_times
The meta key is used set retry times per request. When set, the
`max_retry_times` meta key takes higher precedence over the
`RETRY_TIMES` setting.
#### verbatim_url
> [!WARNING]
> Added in version 2.17.0.
Set this key to `True` to keep the request URL as passed to
`Request`, without URL percent-encoding.
When this key is enabled, `fingerprint()` does not
canonicalize the request URL, so requests whose URLs differ only in
characters that would otherwise be canonicalized get different fingerprints.
In this mode, the `keep_fragments` parameter is ignored, and it is
effectively true.
### Stopping the download of a Response
Raising a `StopDownload` exception from a handler for the
`bytes_received` or `headers_received`
signals will stop the download of a given response. See the following example:
```python
import scrapy
class StopSpider(scrapy.Spider):
name = "stop"
start_urls = ["https://docs.scrapy.org/en/latest/"]
@classmethod
def from_crawler(cls, crawler):
spider = super().from_crawler(crawler)
crawler.signals.connect(
spider.on_bytes_received, signal=scrapy.signals.bytes_received
)
return spider
def parse(self, response):
# 'last_chars' show that the full response was not downloaded
yield {"len": len(response.text), "last_chars": response.text[-40:]}
def on_bytes_received(self, data, request, spider):
raise scrapy.exceptions.StopDownload(fail=False)
```
which produces the following output:
```
2020-05-19 17:26:12 [scrapy.core.engine] INFO: Spider opened
2020-05-19 17:26:12 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2020-05-19 17:26:13 [scrapy.core.downloader.handlers.http11] DEBUG: Download stopped for <GET https://docs.scrapy.org/en/latest/> from signal handler StopSpider.on_bytes_received
2020-05-19 17:26:13 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://docs.scrapy.org/en/latest/> (referer: None) ['download_stopped']
2020-05-19 17:26:13 [scrapy.core.scraper] DEBUG: Scraped from <200 https://docs.scrapy.org/en/latest/>
{'len': 279, 'last_chars': 'dth, initial-scale=1.0">\n \n <title>Scr'}
2020-05-19 17:26:13 [scrapy.core.engine] INFO: Closing spider (finished)
```
By default, resulting responses are handled by their corresponding errbacks. To
call their callback instead, like in this example, pass `fail=False` to the
`StopDownload` exception.
### Request subclasses
Here is the list of built-in `Request` subclasses. You can also subclass
it to implement your own custom functionality.
#### FormRequest
#### *class* scrapy.FormRequest(\*args: Any, \*\*kwargs: Any)
A `Request` subclass with a `formdata` parameter that
url-encodes the given data and assigns it to the request, which makes it
convenient to send arbitrary form data via HTTP POST or GET without an HTML
`<form>` element to parse.
> [!NOTE]
> To build a request from an HTML `<form>` element found in a
> response, use form2request instead. See
> form.
The remaining arguments are the same as for the `Request`
class and are not documented here.
* **Parameters:**
**formdata** (*dict* *or* *collections.abc.Iterable*) – a dictionary (or iterable of (key, value) tuples)
containing HTML form data which will be url-encoded. If
`method` is not given and `formdata` is
provided, the method is set to `"POST"` and the data is assigned to
the request body; if the method is `"GET"`, the data is added to the
URL query string instead.
To send data via HTTP POST, simulating an HTML form submission, return a
`FormRequest` object from your spider:
```python
return [
FormRequest(
url="http://www.example.com/post/action",
formdata={"name": "John Doe", "age": "27"},
callback=self.after_post,
)
]
```
To send the data in the URL query string instead, use the `GET` method:
```python
return [
FormRequest(
url="http://www.example.com/search",
method="GET",
formdata={"q": "keyword", "page": "1"},
callback=self.parse_results,
)
]
```
#### JsonRequest
The JsonRequest class extends the base `Request` class with functionality for
dealing with JSON requests.
#### *class* scrapy.http.JsonRequest([... datadumps_kwargs]url, ,)
The `JsonRequest` class adds two new keyword parameters to the `__init__()` method. The
remaining arguments are the same as for the `Request` class and are
not documented here.
Using the `JsonRequest` will set the `Content-Type` header to `application/json`
and `Accept` header to `application/json, text/javascript, */*; q=0.01`
* **Parameters:**
* **data** (*object*) – is any JSON serializable object that needs to be JSON encoded and assigned to body.
If the `body` argument is provided this parameter will be ignored.
If the `body` argument is not provided and the
`data` argument is provided the `method` will be
set to `'POST'` automatically.
* **dumps_kwargs** (*dict*) – Parameters that will be passed to underlying `json.dumps()` method which is used to serialize
data into JSON format.
##### attributes *: tuple[str, ...]* *= ('url', 'headers', 'body', 'cookies', 'meta', 'encoding', 'flags', 'cb_kwargs', 'callback', 'dont_filter', 'errback', 'method', 'priority', 'dumps_kwargs')*
A tuple of `str` objects containing the name of all public
attributes of the class that are also keyword parameters of the
`__init__()` method.
Currently used by `Request.replace()`, `Request.to_dict()` and
`request_from_dict()`.
#### JsonRequest usage example
Sending a JSON POST request with a JSON payload:
```python
data = {
"name1": "value1",
"name2": "value2",
}
yield JsonRequest(url="http://www.example.com/post/action", data=data)
```
### Response objects
#### *class* scrapy.http.Response(\*args: Any, \*\*kwargs: Any)
An object that represents an HTTP response, which is usually
downloaded (by the Downloader) and fed to the Spiders for processing.
* **Parameters:**
* **url** (*str*) – the URL of this response
* **status** (*int*) – the HTTP status of the response. Defaults to `200`.
* **headers** (*dict*) – the headers of this response. The dict values can be strings
(for single valued headers) or lists (for multi-valued headers).
* **body** (*bytes*) – the response body. To access the decoded text as a string, use
`response.text` from an encoding-aware
Response subclass,
such as `TextResponse`.
* **flags** (*list*) – is a list containing the initial values for the
`Response.flags` attribute. If given, the list will be shallow
copied.
* **request** (*scrapy.Request*) – the initial value of the `Response.request` attribute.
This represents the `Request` that generated this response.
* **certificate** (*Any*) – an object representing the server’s SSL certificate.
* **ip_address** (`ipaddress.IPv4Address` or `ipaddress.IPv6Address`) – The IP address of the server from which the Response originated.
* **protocol** (`str`) – The protocol that was used to download the response.
For instance: “HTTP/1.0”, “HTTP/1.1”, “h2”
##### url
A string containing the URL of the response.
This attribute is read-only. To change the URL of a Response use
`replace()`.
##### status
An integer representing the HTTP status of the response. Example: `200`,
`404`.
##### headers
A dictionary-like (`scrapy.http.headers.Headers`) object which contains
the response headers. Values can be accessed using
`get()` to return the last header value with
the specified name or `getlist()` to return
all header values with the specified name. For example, this call will give you
all cookies in the headers:
```python
response.headers.getlist("Set-Cookie")
```
##### body
The response body as bytes.
If you want the body as a string, use `TextResponse.text` (only
available in `TextResponse` and subclasses).
This attribute is read-only. To change the body of a Response use
`replace()`.
##### request
The `Request` object that generated this response. This attribute is
assigned in the Scrapy engine, after the response and the request have passed
through all Downloader Middlewares.
In particular, this means that:
- HTTP redirections will create a new request from the request before
redirection. It has the majority of the same metadata and original
request attributes and gets assigned to the redirected response
instead of the propagation of the original request.
- Response.request.url doesn’t always equal Response.url
- This attribute is only available in the spider code, and in the
Spider Middlewares, but not in
Downloader Middlewares (although you have the Request available there by
other means) and handlers of the `response_downloaded` signal.
##### meta
A shortcut to the `meta` attribute of the
`Response.request` object (i.e. `self.request.meta`).
Unlike the `Response.request` attribute, the `Response.meta`
attribute is propagated along redirects and retries, so you will get
the original `Request.meta` sent from your spider.
> [!NOTE]
> ### See also
>
> `Request.meta` attribute
##### cb_kwargs
A shortcut to the `cb_kwargs` attribute of the
`Response.request` object (i.e. `self.request.cb_kwargs`).
Unlike the `Response.request` attribute, the
`Response.cb_kwargs` attribute is propagated along redirects and
retries, so you will get the original `Request.cb_kwargs` sent from your spider.
> [!NOTE]
> ### See also
>
> `Request.cb_kwargs` attribute
##### flags
A list that contains flags for this response. Flags are labels used for
tagging Responses. For example: `'cached'`, `'redirected'`’, etc. And
they’re shown on the string representation of the Response (`__str__()`
method) which is used by the engine for logging.
##### certificate
An object representing the server’s SSL certificate. Its type and
contents depend on the download handler that produced the response.
Only populated for `https` responses, `None` otherwise.
##### ip_address
The IP address of the server from which the Response originated.
This attribute is currently only populated by the HTTP download
handlers, i.e. for `http(s)` responses. For other handlers,
`ip_address` is always `None`.
##### protocol
The protocol that was used to download the response.
For instance: “HTTP/1.0”, “HTTP/1.1”
This attribute is currently only populated by the HTTP download
handlers, i.e. for `http(s)` responses. For other handlers,
`protocol` is always `None`.
##### attributes *: tuple[str, ...]* *= ('url', 'headers', 'body', 'flags', 'status', 'request', 'certificate', 'ip_address', 'protocol')*
A tuple of `str` objects containing the name of all public
attributes of the class that are also keyword parameters of the
`__init__()` method.
Currently used by `Response.replace()`.
##### copy()
Returns a new Response which is a copy of this Response.
##### replace([urlstatusheadersbodyrequestflagscertificateip_addressprotocolcls], , , , , , , , ,)
Returns a Response object with the same members, except for those members
given new values by whichever keyword arguments are specified. The
attribute `Response.meta` is copied by default.
##### urljoin(url)
Constructs an absolute url by combining the Response’s `url` with
a possible relative url.
This is a wrapper over `urljoin()`, it’s merely an alias for
making this call:
```python
urllib.parse.urljoin(response.url, url)
```
##### follow(url: str | Link, callback: CallbackT | None = None, method: str = 'GET', headers: Mapping[str, Any] | Mapping[bytes, Any] | Iterable[tuple[str | bytes, Any]] | None = None, body: bytes | str | None = None, cookies: CookiesT | None = None, meta: dict[str, Any] | None = None, encoding: str | None = 'utf-8', priority: int = 0, dont_filter: bool = False, errback: Callable[[Failure], Any] | None = None, cb_kwargs: dict[str, Any] | None = None, flags: list[str] | None = None) → Request
Return a `Request` instance to follow a link `url`.
It accepts the same arguments as `Request.__init__()` method,
but `url` can be a relative URL or a `Link` object,
not only an absolute URL.
`TextResponse` provides a `follow()`
method which supports selectors in addition to absolute/relative URLs
and Link objects.
##### follow_all(urls: Iterable[str | Link], callback: CallbackT | None = None, method: str = 'GET', headers: Mapping[str, Any] | Mapping[bytes, Any] | Iterable[tuple[str | bytes, Any]] | None = None, body: bytes | str | None = None, cookies: CookiesT | None = None, meta: dict[str, Any] | None = None, encoding: str | None = 'utf-8', priority: int = 0, dont_filter: bool = False, errback: Callable[[Failure], Any] | None = None, cb_kwargs: dict[str, Any] | None = None, flags: list[str] | None = None) → Iterable[Request]
Return an iterable of `Request` instances to follow all links
in `urls`. It accepts the same arguments as `Request.__init__()` method,
but elements of `urls` can be relative URLs or `Link` objects,
not only absolute URLs.
`TextResponse` provides a `follow_all()`
method which supports selectors in addition to absolute/relative URLs
and Link objects.
> [!CAUTION]
> Every returned request gets its own *meta* and
> *cb_kwargs* dictionaries, but the values within them are shared.
> Mutating one of those values, e.g. appending to a list, affects
> all the returned requests.
##### to_dict() → dict[str, Any]
Return a dictionary containing the Response’s data.
> [!WARNING]
> Added in version 2.19.0.
Use `response_from_dict()` to convert back
into a `Response` object.
`request` and `certificate` are left out, as they are tied
to a single crawl. Everything else in `attributes` is included,
so subclasses only need to override this method, and `from_dict()`,
if some of their attributes cannot be stored as is.
##### *classmethod* from_dict(d: dict[str, Any]) → Self
Return a response built from the *d* dict, as returned by
`to_dict()`.
> [!WARNING]
> Added in version 2.19.0.
### Response subclasses
Here is the list of available built-in Response subclasses. You can also
subclass the Response class to implement your own functionality.
#### TextResponse objects
#### *class* scrapy.http.TextResponse([encoding[...]]url, ,)
`TextResponse` objects adds encoding capabilities to the base
`Response` class, which is meant to be used only for binary data,
such as images, sounds or any media file.
`TextResponse` objects support a new `__init__()` method argument, in
addition to the base `Response` objects. The remaining functionality
is the same as for the `Response` class and is not documented here.
* **Parameters:**
**encoding** (*str*) – is a string which contains the encoding to use for this
response. If you create a `TextResponse` object with a string as
body, it will be converted to bytes encoded using this encoding. If
*encoding* is `None` (default), the encoding will be looked up in the
response headers and body instead.
`TextResponse` objects support the following attributes in addition
to the standard `Response` ones:
##### text
Response body, as a string.
The same as `response.body.decode(response.encoding)`, but the
result is cached after the first call, so you can access
`response.text` multiple times without extra overhead.
> [!NOTE]
> `str(response.body)` is not a correct way to convert the response
> body into a string:
>
> ```pycon
> >>> str(b"body")
> "b'body'"
> ```
##### encoding
A string with the encoding of this response. The encoding is resolved by
trying the following mechanisms, in order:
1. the encoding passed in the `__init__()` method `encoding` argument
2. the encoding of the [byte order mark](https://en.wikipedia.org/wiki/Byte_order_mark) at the start of the response
body
3. the encoding declared in the Content-Type HTTP header. If this
encoding is not valid (i.e. unknown), it is ignored and the next
resolution mechanism is tried.
4. the encoding declared in the response body. The TextResponse class
doesn’t provide any special functionality for this. However, the
`HtmlResponse` and `XmlResponse` classes do.
5. the encoding inferred by looking at the response body. This is the more
fragile method but also the last one tried.
This order matches the [encoding sniffing algorithm](https://html.spec.whatwg.org/multipage/parsing.html#determining-the-character-encoding) of the HTML
standard, which web browsers follow.
To resolve the encoding differently, determine it yourself and pass it
through `Response.replace()` from a downloader middleware. Give that middleware an order between
those of
`MetaRefreshMiddleware`
(580) and
`HttpCompressionMiddleware`
(590), so that it gets a decompressed body and no other component reads
the response text before it.
For example, to give a declaration in the response body precedence over
the Content-Type header:
```python
from w3lib.encoding import html_body_declared_encoding, read_bom
from scrapy.http import TextResponse
class BodyEncodingMiddleware:
def process_response(self, request, response, spider):
if not isinstance(response, TextResponse):
return response
if read_bom(response.body)[0]:
return response
encoding = html_body_declared_encoding(response.body)
return response.replace(encoding=encoding) if encoding else response
```
##### selector
A `Selector` instance using the response as
target. The selector is lazily instantiated on first access.
##### attributes *: tuple[str, ...]* *= ('url', 'headers', 'body', 'flags', 'status', 'request', 'certificate', 'ip_address', 'protocol', 'encoding')*
A tuple of `str` objects containing the name of all public
attributes of the class that are also keyword parameters of the
`__init__()` method.
Currently used by `Response.replace()`.
`TextResponse` objects support the following methods in addition to
the standard `Response` ones:
##### jmespath(query)
A shortcut to `TextResponse.selector.jmespath(query)`:
```python
response.jmespath("object.[*]")
```
##### xpath(query)
A shortcut to `TextResponse.selector.xpath(query)`:
```python
response.xpath("//p")
```
##### css(query)
A shortcut to `TextResponse.selector.css(query)`:
```python
response.css("p")
```
##### follow(url: str | Link | parsel.Selector, callback: CallbackT | None = None, method: str = 'GET', headers: Mapping[str, Any] | Mapping[bytes, Any] | Iterable[tuple[str | bytes, Any]] | None = None, body: bytes | str | None = None, cookies: CookiesT | None = None, meta: dict[str, Any] | None = None, encoding: str | None = None, priority: int = 0, dont_filter: bool = False, errback: Callable[[Failure], Any] | None = None, cb_kwargs: dict[str, Any] | None = None, flags: list[str] | None = None) → Request
Return a `Request` instance to follow a link `url`.
It accepts the same arguments as `Request.__init__()` method,
but `url` can be not only an absolute URL, but also
* a relative URL
* a `Link` object, e.g. the result of
topics-link-extractors
* a `Selector` object for a `<link>` or `<a>` element, e.g.
`response.css('a.my_link')[0]`
* an attribute `Selector` (not SelectorList), e.g.
`response.css('a::attr(href)')[0]` or
`response.xpath('//img/@src')[0]`
See response-follow-example for usage examples.
##### follow_all(urls: Iterable[str | Link] | parsel.SelectorList[Any] | None = None, callback: CallbackT | None = None, method: str = 'GET', headers: Mapping[str, Any] | Mapping[bytes, Any] | Iterable[tuple[str | bytes, Any]] | None = None, body: bytes | str | None = None, cookies: CookiesT | None = None, meta: dict[str, Any] | None = None, encoding: str | None = None, priority: int = 0, dont_filter: bool = False, errback: Callable[[Failure], Any] | None = None, cb_kwargs: dict[str, Any] | None = None, flags: list[str] | None = None, css: str | None = None, xpath: str | None = None) → Iterable[Request]
A generator that produces `Request` instances to follow all
links in `urls`. It accepts the same arguments as the `Request`’s
`__init__()` method, except that each `urls` element does not need to be
an absolute URL, it can be any of the following:
* a relative URL
* a `Link` object, e.g. the result of
topics-link-extractors
* a `Selector` object for a `<link>` or `<a>` element, e.g.
`response.css('a.my_link')[0]`
* an attribute `Selector` (not SelectorList), e.g.
`response.css('a::attr(href)')[0]` or
`response.xpath('//img/@src')[0]`
In addition, `css` and `xpath` arguments are accepted to perform the link extraction
within the `follow_all()` method (only one of `urls`, `css` and `xpath` is accepted).
Note that when passing a `SelectorList` as argument for the `urls` parameter or
using the `css` or `xpath` parameters, this method will not produce requests for
selectors from which links cannot be obtained (for instance, anchor tags without an
`href` attribute)
> [!NOTE]
> ### See also
>
> `Response.follow_all()`, for a caution about mutable
> *meta* and *cb_kwargs* values.
##### json()
Deserialize a JSON document to a Python object.
> [!WARNING]
> Changed in version 2.18.0: Bodies that cannot be decoded as UTF-8, UTF-16 or UTF-32, as the
> JSON specification requires, are now decoded using
> `TextResponse.encoding` instead of raising
> `UnicodeDecodeError`.
The result is cached after the first call.
##### urljoin(url)
Constructs an absolute url by combining the Response’s base url with
a possible relative url. The base url shall be extracted from the
`<base>` tag, or just `Response.url` if there is no such
tag.
#### HtmlResponse objects
#### *class* scrapy.http.HtmlResponse([...]url,)
The `HtmlResponse` class is a subclass of `TextResponse`
which adds encoding auto-discovering support by looking into the HTML [meta
http-equiv](https://www.w3schools.com/TAGS/att_meta_http_equiv.asp) attribute. See `TextResponse.encoding`.
#### XmlResponse objects
#### *class* scrapy.http.XmlResponse([...]url,)
The `XmlResponse` class is a subclass of `TextResponse` which
adds encoding auto-discovering support by looking into the XML declaration
line. See `TextResponse.encoding`.
#### JsonResponse objects
#### *class* scrapy.http.JsonResponse([...]url,)
The `JsonResponse` class is a subclass of `TextResponse`
that is used when the response has a [JSON MIME type](https://mimesniff.spec.whatwg.org/#json-mime-type) in its Content-Type
header.
### Other functions related to responses
#### scrapy.utils.response.response_from_dict(d: dict[str, Any]) → Response
Return a response built from the *d* dict, as returned by
`Response.to_dict()`.
> [!WARNING]
> Added in version 2.19.0.
If *d* does not indicate a response class, e.g. because it comes from a
plain `Response` object or predates `to_dict()`,
the class is guessed with `responsetypes`.
Source: /topics/cookies.md
## Cookies
Scrapy keeps track of the cookies that websites set and sends them back on
later requests to those websites, just like a web browser does. That is the job
of `CookiesMiddleware`, which is
enabled by default.
### Setting cookies on a request
Use the `cookies` parameter of `Request` to send cookies of
your own, either as a dict:
```python
request = Request(
url="https://example.com",
cookies={"currency": "USD", "country": "UY"},
)
```
Or as a list of dicts, which also lets you set cookie attributes:
```python
request = Request(
url="https://example.com",
cookies=[
{
"name": "currency",
"value": "USD",
"domain": "example.com",
"path": "/currency",
"secure": True,
},
],
)
```
Setting attributes is only useful if the cookies are stored for later requests,
i.e. if `dont_merge_cookies` is not enabled.
> [!CAUTION]
> Cookies set through the `Cookie` header are not handled by
> `CookiesMiddleware`, which
> drops that header.
> [!CAUTION]
> When a cookie name or value is a byte sequence that is not UTF-8
> encoded, the cookie is dropped and a warning is logged. See
> topics-logging-advanced-customization to customize the logging
> behavior.
### Multiple cookie sessions per spider
By default all requests share a single cookie jar (session). To use different
ones, pass an identifier in the `cookiejar` request meta key:
```python
for i, url in enumerate(urls):
yield Request(url, meta={"cookiejar": i}, callback=self.parse_page)
```
The `cookiejar` meta key is not “sticky”, so you need to keep passing
it along on subsequent requests:
```python
def parse_page(self, response):
return Request(
"https://example.com/otherpage",
meta={"cookiejar": response.meta["cookiejar"]},
callback=self.parse_other_page,
)
```
### Skipping the cookie jar for a request
Set the `dont_merge_cookies` request meta key to `True` to keep a
request from touching the cookie jar in either direction: no stored cookie is
sent with the request, and no cookie received in the response is stored. The
cookies of the request itself are ignored as well.
### COOKIES_ENABLED
Default: `True`
Whether to enable `CookiesMiddleware`.
If disabled, no cookies are sent to web servers.
### COOKIES_DEBUG
Default: `False`
If enabled, Scrapy logs all cookies sent in requests (i.e. the `Cookie`
header) and all cookies received in responses (i.e. the `Set-Cookie`
header):
```
2011-04-06 14:35:10-0300 [scrapy.core.engine] INFO: Spider opened
2011-04-06 14:35:10-0300 [scrapy.downloadermiddlewares.cookies] DEBUG: Sending cookies to: <GET http://www.diningcity.com/netherlands/index.html>
Cookie: clientlanguage_nl=en_EN
2011-04-06 14:35:14-0300 [scrapy.downloadermiddlewares.cookies] DEBUG: Received cookies from: <200 http://www.diningcity.com/netherlands/index.html>
Set-Cookie: JSESSIONID=B~FA4DC0C496C8762AE4F1A620EAB34F38; Path=/
Set-Cookie: ip_isocode=US
Set-Cookie: clientlanguage_nl=en_EN; Expires=Thu, 07-Apr-2011 21:21:34 GMT; Path=/
2011-04-06 14:49:50-0300 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.diningcity.com/netherlands/index.html> (referer: None)
[...]
```
### CookiesMiddleware
#### *class* scrapy.downloadermiddlewares.cookies.CookiesMiddleware(debug: bool = False)
This middleware enables working with sites that need cookies
Source: /topics/link-extractors.md
## Link Extractors
A link extractor is an object that extracts links from responses.
The `__init__` method of
`LxmlLinkExtractor` takes settings that
determine which links may be extracted. `LxmlLinkExtractor.extract_links` returns a
list of matching `Link` objects from a
`Response` object.
Link extractors are used in `CrawlSpider` spiders
through a set of `Rule` objects.
You can also use link extractors in regular spiders. For example, you can instantiate
`LinkExtractor` into a class
variable in your spider, and use it from your spider callbacks:
```python
def parse(self, response):
for link in self.link_extractor.extract_links(response):
yield Request(link.url, callback=self.parse)
```
### Link extractor reference
The link extractor class is
`scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor`. For convenience it
can also be imported as `scrapy.linkextractors.LinkExtractor`:
```python
from scrapy.linkextractors import LinkExtractor
```
#### LxmlLinkExtractor
#### *class* scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor(allow: str | Pattern[str] | Iterable[str | Pattern[str]] = (), deny: str | Pattern[str] | Iterable[str | Pattern[str]] = (), allow_domains: str | Iterable[str] = (), deny_domains: str | Iterable[str] = (), restrict_xpaths: str | Iterable[str] = (), tags: str | Iterable[str] | None = ('a', 'area'), attrs: str | Iterable[str] | None = ('href',), canonicalize: bool = False, unique: bool = True, process_value: Callable[[Any], Any] | None = None, deny_extensions: str | Iterable[str] | None = None, restrict_css: str | Iterable[str] = (), strip: bool = True, restrict_text: str | Pattern[str] | Iterable[str | Pattern[str]] | None = None, deny_tags: str | Iterable[str] = (), deny_attrs: str | Iterable[str] = ())
LxmlLinkExtractor is the recommended link extractor with handy filtering
options. It is implemented using lxml’s robust HTMLParser.
* **Parameters:**
* **allow** (*str* *or* *list*) – a single regular expression (or list of regular expressions)
that the (absolute) urls must match in order to be extracted. If not
given (or empty), it will match all links.
* **deny** (*str* *or* *list*) – a single regular expression (or list of regular expressions)
that the (absolute) urls must match in order to be excluded (i.e. not
extracted). It has precedence over the `allow` parameter. If not
given (or empty) it won’t exclude any links.
* **allow_domains** (*str* *or* *list*) – a single value or a list of string containing
domains which will be considered for extracting the links
* **deny_domains** (*str* *or* *list*) – a single value or a list of strings containing
domains which won’t be considered for extracting the links
* **deny_extensions** (*list*) – a single value or list of strings containing
extensions that should be ignored when extracting links.
If not given, it will default to
`scrapy.linkextractors.IGNORED_EXTENSIONS`.
* **restrict_xpaths** (*str* *or* *list*) – is an XPath (or list of XPath’s) which defines
regions inside the response where links should be extracted from.
If given, only the text selected by those XPath will be scanned for
links.
* **restrict_css** (*str* *or* *list*) – a CSS selector (or list of selectors) which defines
regions inside the response where links should be extracted from.
Has the same behaviour as `restrict_xpaths`.
* **restrict_text** (*str* *or* *list*) – a single regular expression (or list of regular
expressions) that the link’s text must match in order to be extracted.
If not given (or empty), it will match all links. If a list of regular
expressions is given, the link will be extracted if it matches at least
one.
* **tags** (*str* *or* *list*) – a tag or a list of tags to consider when extracting links.
Defaults to `('a', 'area')`. Use `'*'` to consider every tag.
* **attrs** (*list*) – an attribute or list of attributes which should be considered
when looking for links to extract (only for those tags specified in the
`tags` parameter). Defaults to `('href',)`. Use `'*'` to consider
every attribute.
* **deny_tags** (*str* *or* *list*) –
a tag or a list of tags that should not be considered when
extracting links. It has precedence over the `tags` parameter, so it
can be combined with `tags='*'` to consider every tag except a few.
Defaults to `()` (no tag is excluded).
> [!WARNING]
> Added in version 2.17.0.
* **deny_attrs** (*str* *or* *list*) –
an attribute or a list of attributes that should not be
considered when looking for links to extract. It has precedence over the
`attrs` parameter, so it can be combined with `attrs='*'` to consider
every attribute except a few. Defaults to `()` (no attribute is
excluded).
> [!WARNING]
> Added in version 2.17.0.
* **canonicalize** (*bool*) – canonicalize each extracted url (using
w3lib.url.canonicalize_url). Defaults to `False`.
Note that canonicalize_url is meant for duplicate checking;
it can change the URL visible at server side, so the response can be
different for requests with canonicalized and raw URLs. If you’re
using LinkExtractor to follow links it is more robust to
keep the default `canonicalize=False`.
* **unique** (*bool*) – whether duplicate filtering should be applied to extracted
links.
* **process_value** (*collections.abc.Callable*) –
a function which receives each value extracted from
the tag and attributes scanned and can modify the value and return a
new one, or return `None` to ignore the link altogether. If not
given, `process_value` defaults to `lambda x: x`.
For example, to extract links from this code:
```html
<a href="javascript:goToPage('../other/page.html'); return false">Link text</a>
```
You can use the following function in `process_value`:
```python
def process_value(value):
m = re.search(r"javascript:goToPage\('(.*?)'", value)
if m:
return m.group(1)
```
`process_value` is called before the filtering parameters, such as
`allow` and `deny`, which match the value that it returns. To drop
links based on their final URL, use the `process_links` parameter of
`Rule`, which only receives links that those
parameters kept.
* **strip** (*bool*) – whether to strip whitespaces from extracted attributes.
According to HTML5 standard, leading and trailing whitespaces
must be stripped from `href` attributes of `<a>`, `<area>`
and many other elements, `src` attribute of `<img>`, `<iframe>`
elements, etc., so LinkExtractor strips space chars by default.
Set `strip=False` to turn it off (e.g. if you’re extracting urls
from elements or attributes which allow leading/trailing whitespaces).
##### extract_links(response: TextResponse) → list[Link]
Returns a list of `Link` objects from the
specified `response`.
Only links that match the settings passed to the `__init__` method of
the link extractor are returned.
Duplicate links are omitted if the `unique` attribute is set to `True`,
otherwise they are returned.
#### Link
#### *class* scrapy.link.Link(url: str, text: str = '', fragment: str = '', nofollow: bool = False)
Link objects represent an extracted link by the LinkExtractor.
Using the anchor tag sample below to illustrate the parameters:
```html
<a href="https://example.com/nofollow.html#foo" rel="nofollow">Dont follow this one</a>
```
* **Parameters:**
* **url** – the absolute url being linked to in the anchor tag.
From the sample, this is `https://example.com/nofollow.html`.
* **text** – the text in the anchor tag. From the sample, this is `Dont follow this one`.
* **fragment** – the part of the url after the hash symbol. From the sample, this is `foo`.
* **nofollow** – an indication of the presence or absence of a nofollow value in the `rel` attribute
of the anchor tag.
Source: /topics/settings.md
## Settings
The Scrapy settings allows you to customize the behaviour of all Scrapy
components, including the core, extensions, pipelines and spiders themselves.
The infrastructure of the settings provides a global namespace of key-value mappings
that the code can use to pull configuration values from. The settings can be
populated through different mechanisms, which are described below.
The settings are also the mechanism for selecting the currently active Scrapy
project (in case you have many).
For a list of available built-in settings see: topics-settings-ref.
### Designating the settings
When you use Scrapy, you have to tell it which settings you’re using. You can
do this by using an environment variable, `SCRAPY_SETTINGS_MODULE`.
The value of `SCRAPY_SETTINGS_MODULE` should be in Python path syntax, e.g.
`myproject.settings`. Note that the settings module should be on the
Python import search path.
### Populating the settings
Settings can be populated using different mechanisms, each of which has a
different precedence:
> 1. Command-line settings (highest precedence)
> 2. Spider settings
> 3. Project settings
> 4. Add-on settings
> 5. Command-specific default settings
> 6. Global default settings (lowest precedence)
#### 1. Command-line settings
Settings set in the command line have the highest precedence, overriding any
other settings.
You can explicitly override one or more settings using the `-s` (or
`--set`) command-line option.
Example:
```sh
scrapy crawl myspider -s LOG_LEVEL=INFO -s LOG_FILE=scrapy.log
```
#### 2. Spider settings
Spiders can define their own settings that will take
precedence and override the project ones.
> [!NOTE]
> Pre-crawler settings cannot be defined
> per spider, and reactor settings and
> logging settings are subject to restrictions when
> running multiple spiders in the same process.
One way to do so is by setting their `custom_settings`
attribute:
```python
import scrapy
class MySpider(scrapy.Spider):
name = "myspider"
custom_settings = {
"SOME_SETTING": "some value",
}
```
It’s often better to implement `update_settings()` instead,
and settings set there should use the `"spider"` priority explicitly:
```python
import scrapy
class MySpider(scrapy.Spider):
name = "myspider"
@classmethod
def update_settings(cls, settings):
super().update_settings(settings)
settings.set("SOME_SETTING", "some value", priority="spider")
```
> [!WARNING]
> Added in version 2.11.
It’s also possible to modify the settings in the
`from_crawler()` method, e.g. based on spider
arguments or other logic:
```python
import scrapy
class MySpider(scrapy.Spider):
name = "myspider"
@classmethod
def from_crawler(cls, crawler, *args, **kwargs):
spider = super().from_crawler(crawler, *args, **kwargs)
if "some_argument" in kwargs:
spider.settings.set(
"SOME_SETTING", kwargs["some_argument"], priority="spider"
)
return spider
```
#### 3. Project settings
Scrapy projects include a settings module, usually a file called
`settings.py`, where you should populate most settings that apply to all your
spiders.
`scrapy.utils.project.get_project_settings()` returns these settings, e.g.
to pass them to `AsyncCrawlerProcess` when running
Scrapy from a script.
> [!NOTE]
> ### See also
>
> topics-settings-module-envvar
#### 4. Add-on settings
Add-ons can modify settings. They should do this with
`"addon"` priority where possible.
#### 5. Command-specific default settings
Each Scrapy command can have its own default settings,
which override the global default settings.
Those command-specific default settings are specified in the
`default_settings` attribute of each command class.
#### 6. Default global settings
The `scrapy.settings.default_settings` module defines global default values
for some built-in settings.
> [!NOTE]
> `startproject` generates a `settings.py` file that sets
> some settings to different values.
>
> The reference documentation of settings indicates the default value if one
> exists. If `startproject` sets a value, that value is documented
> as default, and the value from `scrapy.settings.default_settings` is
> documented as “fallback”.
### Compatibility with pickle
Setting values must be picklable.
### Import paths and classes
When a setting references a callable object to be imported by Scrapy, such as a
class or a function, there are two different ways you can specify that object:
- As a string containing the import path of that object
- As the object itself
For example:
```python
from mybot.pipelines.validate import ValidateMyItem
ITEM_PIPELINES = {
# passing the classname...
ValidateMyItem: 300,
# ...equals passing the class path
"mybot.pipelines.validate.ValidateMyItem": 300,
}
```
> [!NOTE]
> Passing non-callable objects is not supported.
### How to access settings
In a spider, settings are available through `self.settings`:
```python
class MySpider(scrapy.Spider):
name = "myspider"
start_urls = ["http://example.com"]
def parse(self, response):
print(f"Existing settings: {self.settings.attributes.keys()}")
```
> [!NOTE]
> The `settings` attribute is set in the base Spider class after the spider
> is initialized. If you want to use settings before the initialization
> (e.g., in your spider’s `__init__()` method), you’ll need to override the
> `from_crawler()` method.
Components can also access settings.
The `settings` object can be used like a `dict` (e.g.
`settings["LOG_ENABLED"]`). However, to support non-string setting values,
which may be passed from the command line as strings, it is recommended to use
one of the methods provided by the `Settings` API.
### Component priority dictionaries
A **component priority dictionary** is a `dict` where keys are
components and values are component priorities. For
example:
```python
{
"path.to.ComponentA": None,
ComponentB: 100,
}
```
A component can be specified either as a class object or through an import
path.
> [!WARNING]
> Component priority dictionaries are regular `dict` objects.
> Be careful not to define the same component more than once, e.g. with
> different import path strings or defining both an import path and a
> `type` object.
A priority can be an `int` or `None`.
A component with priority 1 goes *before* a component with priority 2. What
going before entails, however, depends on the corresponding setting. For
example, in the `DOWNLOADER_MIDDLEWARES` setting, components have
their
`process_request()`
method executed before that of later components, but have their
`process_response()`
method executed after that of later components.
A component with priority `None` is disabled.
Some component priority dictionaries get merged with some built-in value. For
example, `DOWNLOADER_MIDDLEWARES` is merged with
`DOWNLOADER_MIDDLEWARES_BASE`. This is where `None` comes in
handy, allowing you to disable a component from the base setting in the regular
setting:
```python
DOWNLOADER_MIDDLEWARES = {
"scrapy.downloadermiddlewares.offsite.OffsiteMiddleware": None,
}
```
### Special settings
The following settings work slightly differently than all other settings.
#### Pre-crawler settings
**Pre-crawler settings** are settings used before the
`Crawler` object is created.
These settings cannot be set from a spider.
These settings are:
- `ADDONS`
- `COMMANDS_MODULE`
- `FORCE_CRAWLER_PROCESS`
- `SPIDER_LOADER_CLASS` and settings used by the corresponding
spider loader class, e.g. `SPIDER_MODULES` and
`SPIDER_LOADER_WARN_ONLY` for the default spider loader class.
- `TWISTED_REACTOR_ENABLED`
`ADDONS` is a special case: it can be set from a spider, but the
`update_pre_crawler_settings()` method of add-ons
enabled that way is not called.
`TWISTED_REACTOR` also acts as a pre-crawler setting when running a
command that needs a CrawlerProcess,
since its project-level value determines the crawler process class.
#### Reactor settings
**Reactor settings** are settings tied to the Twisted reactor.
Because only 1 reactor can be used per process, these settings cannot use a
different value per spider when running multiple spiders in the same
process.
These settings are used upon installing the reactor:
- `ASYNCIO_EVENT_LOOP` (not possible to set per-spider when using
`AsyncCrawlerProcess`, see below)
- `TWISTED_REACTOR` (ignored when using
`AsyncCrawlerProcess`, see below)
They can be set from a spider, but only the values
from the first spider that runs are used, since that is when the reactor is
installed. If a later spider asks for a different reactor or a different event
loop, an exception is raised. With
`CrawlerRunner` and
`AsyncCrawlerRunner` the reactor must be installed
beforehand, so these settings are only used to check that the installed reactor
and event loop match them.
These settings are applied when starting the reactor:
- `TWISTED_DNS_RESOLVER` and settings used by the corresponding
component, e.g. `DNSCACHE_ENABLED`, `DNSCACHE_SIZE`
and `DNS_TIMEOUT` for the default one.
- `REACTOR_THREADPOOL_MAXSIZE`
They are read from the settings of the
`CrawlerProcess` or
`AsyncCrawlerProcess` object, so setting them from a
spider or an add-on has no effect. They are ignored
altogether when using `CrawlerRunner` or
`AsyncCrawlerRunner`, which do not start the reactor.
There is an additional restriction for `TWISTED_REACTOR` and
`ASYNCIO_EVENT_LOOP` when using
`AsyncCrawlerProcess`: when this class is instantiated,
it installs `AsyncioSelectorReactor`,
ignoring the value of `TWISTED_REACTOR` and using the value of
`ASYNCIO_EVENT_LOOP` that was passed to
`AsyncCrawlerProcess.__init__()`. If a different value for
`TWISTED_REACTOR` or `ASYNCIO_EVENT_LOOP` is provided later,
e.g. in per-spider settings, an exception will be
raised.
All of these settings, except for `ASYNCIO_EVENT_LOOP`, are only used
when the Twisted reactor is used, i.e. when `TWISTED_REACTOR_ENABLED`
is `True`.
#### Logging settings
**Logging settings** are settings that configure the global root logging
handler installed by `configure_logging()`.
These settings can be defined from a spider. However, because only 1 root
logging handler is active per process, these settings cannot use a different
value per spider when running multiple spiders in the same process.
These settings are:
- `LOG_COLOR`
- `LOG_DATEFORMAT`
- `LOG_ENABLED`
- `LOG_ENCODING`
- `LOG_FILE`
- `LOG_FILE_APPEND`
- `LOG_FORMAT`
- `LOG_INSTALL_ROOT_HANDLER`
- `LOG_LEVEL`
- `LOG_SHORT_NAMES`
- `LOG_STDOUT`
### Built-in settings reference
Here’s a list of all available Scrapy settings, in alphabetical order, along
with their default values and the scope where they apply.
The scope, where available, shows where the setting is being used, if it’s tied
to any particular component. In that case the module of that component will be
shown, typically an extension, middleware or pipeline. It also means that the
component must be enabled in order for the setting to have any effect.
#### ADDONS
Default: `{}`
A dict containing paths to the add-ons enabled in your project and their
priorities. For more information, see topics-addons.
> [!NOTE]
> This is a pre-crawler setting, with a
> caveat described in that section.
#### ASYNCIO_EVENT_LOOP
Default: `None`
Import path of a given `asyncio` event loop class.
If the asyncio reactor is enabled (see `TWISTED_REACTOR`) or when
running Scrapy without a reactor this setting
can be used to specify the
asyncio event loop to be used with it. Set the setting to the import path of the
desired asyncio event loop class. If the setting is set to `None` the default asyncio
event loop will be used.
If you are installing the asyncio reactor manually using the `install_reactor()`
function, you can use the `event_loop_path` parameter to indicate the import path of the event loop
class to be used.
Note that the event loop class must inherit from `asyncio.AbstractEventLoop`.
> [!CAUTION]
> Please be aware that, when using a non-default event loop
> (either defined via `ASYNCIO_EVENT_LOOP` or installed with
> `install_reactor()`), Scrapy will call
> `asyncio.set_event_loop()`, which will set the specified event loop
> as the current loop for the current OS thread.
> [!NOTE]
> This is a reactor setting.
#### AWS_ACCESS_KEY_ID
Default: `None`
The AWS access key used by code that requires access to [Amazon Web services](https://aws.amazon.com/),
such as the S3 feed storage backend.
#### AWS_ENDPOINT_URL
Default: `None`
Endpoint URL used for S3-like storage, for example Minio or s3.scality.
#### AWS_MAX_POOL_CONNECTIONS
> [!WARNING]
> Added in version 2.18.0.
Default: `None`
Maximum number of connections that AWS clients, such as those of the
S3 feed storage backend and of the
S3 media pipeline storage backend, keep in their
connection pool.
If `None`, the value of `REACTOR_THREADPOOL_MAXSIZE` is used.
Values lower than the number of parallel AWS calls do not limit those calls, but
their connections are closed instead of reused, which hurts performance, and
`Connection pool is full, discarding connection` warnings are logged.
#### AWS_REGION_NAME
Default: `None`
The name of the region associated with the AWS client.
#### AWS_SECRET_ACCESS_KEY
Default: `None`
The AWS secret key used by code that requires access to [Amazon Web services](https://aws.amazon.com/),
such as the S3 feed storage backend.
#### AWS_SESSION_TOKEN
Default: `None`
The AWS security token used by code that requires access to [Amazon Web services](https://aws.amazon.com/),
such as the S3 feed storage backend, when using
[temporary security credentials](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html).
#### AWS_USE_SSL
Default: `None`
Use this option if you want to disable SSL connection for communication with
S3 or S3-like storage. By default SSL will be used.
#### AWS_VERIFY
Default: `None`
Verify SSL connection between Scrapy and S3 or S3-like storage. By default
SSL verification will occur.
#### BOT_NAME
Default: `<project name>` (fallback: `'scrapybot'`)
The name of the bot implemented by this Scrapy project (also known as the
project name). This name will be used for the logging too.
It’s automatically populated with your project name when you create your
project with the `startproject` command.
#### CONCURRENT_ITEMS
Default: `100`
Maximum number of concurrent items (per response) to process in parallel in
item pipelines.
#### CONCURRENT_REQUESTS
Default: `16`
The maximum number of concurrent (i.e. simultaneous) requests that will be
performed by the Scrapy downloader. Use `0` for no limit.
#### CONCURRENT_REQUESTS_PER_DOMAIN
Default: `1` (fallback: `8`)
The maximum number of concurrent (i.e. simultaneous) requests that will be
performed to any single domain.
See also: topics-autothrottle and its
`AUTOTHROTTLE_TARGET_CONCURRENCY` option.
It is possible to change this setting per domain by using
`DOWNLOAD_SLOTS`.
#### DEFAULT_DROPITEM_LOG_LEVEL
Default: `"WARNING"`
Default log level of messages about dropped items.
When an item is dropped by raising `scrapy.exceptions.DropItem` from the
`process_item()` method of an item pipeline,
a message is logged, and by default its log level is the one configured in this
setting.
You may specify this log level as an integer (e.g. `20`), as a log level
constant (e.g. `logging.INFO`) or as a string with the name of a log level
constant (e.g. `"INFO"`).
When writing an item pipeline, you can force a different log level by setting
`scrapy.exceptions.DropItem.log_level` in your
`scrapy.exceptions.DropItem` exception. For example:
```python
from scrapy.exceptions import DropItem
class MyPipeline:
def process_item(self, item):
if not item.get("price"):
raise DropItem("Missing price data", log_level="INFO")
return item
```
#### DEFAULT_ITEM_CLASS
Default: `'scrapy.item.Item'`
The default class that will be used for instantiating items in the the
Scrapy shell.
#### DEFAULT_REQUEST_HEADERS
Default:
```python
{
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"Accept-Language": "en",
}
```
The default headers used for Scrapy HTTP Requests. They’re populated in the
`DefaultHeadersMiddleware`.
> [!CAUTION]
> Cookies set via the `Cookie` header are not considered by the
> cookie middleware. If you need to set cookies for a
> request, use the `Request.cookies` parameter.
> [!CAUTION]
> A `Referer` header defined here only reaches requests for which
> `RefererMiddleware` does not set
> one, such as start requests. To send it on every request, set
> `REFERRER_POLICY` to `"no-referrer"`.
#### DEPTH_LIMIT
Default: `0`
Scope: `scrapy.spidermiddlewares.depth.DepthMiddleware`
The maximum depth that will be allowed to crawl for any site. If zero, no limit
will be imposed.
#### DEPTH_PRIORITY
Default: `0`
Scope: `scrapy.spidermiddlewares.depth.DepthMiddleware`
An integer that is used to adjust the `priority` of
a `Request` based on its depth.
The priority of a request is adjusted as follows:
```python
request.priority = request.priority - (depth * DEPTH_PRIORITY)
```
As depth increases, positive values of `DEPTH_PRIORITY` decrease request
priority (BFO), while negative values increase request priority (DFO). See
also faq-bfo-dfo.
> [!NOTE]
> This setting adjusts priority **in the opposite way** compared to
> other priority settings `REDIRECT_PRIORITY_ADJUST`
> and `RETRY_PRIORITY_ADJUST`.
#### DEPTH_STATS_VERBOSE
Default: `False`
Scope: `scrapy.spidermiddlewares.depth.DepthMiddleware`
Whether to collect verbose depth stats. If this is enabled, the number of
requests for each depth is collected in the stats.
#### DNSCACHE_ENABLED
Default: `True`
Whether to enable DNS in-memory cache.
> [!NOTE]
> This setting is only used by
> `CachingThreadedResolver` and
> `CachingHostnameResolver`. It has no effect when
> `TWISTED_REACTOR_ENABLED` is `False`, and may have no effect
> either when `TWISTED_DNS_RESOLVER` is set to a different resolver.
> [!NOTE]
> This is a reactor setting.
#### DNSCACHE_SIZE
Default: `10000`
DNS in-memory cache size, see `DNSCACHE_ENABLED`.
> [!NOTE]
> This is a reactor setting.
#### DNS_TIMEOUT
Default: `60`
Timeout for processing of DNS queries in seconds. Float is supported.
The timeout starts when the query is queued into the Twisted reactor thread
pool, not when it is sent. If that thread pool is saturated, queries can time
out before being sent, in which case increasing
`REACTOR_THREADPOOL_MAXSIZE` helps more than increasing this setting.
> [!NOTE]
> This setting is only used by
> `CachingThreadedResolver`. It has no effect when
> `TWISTED_REACTOR_ENABLED` is `False`, and may have no effect
> either when `TWISTED_DNS_RESOLVER` is set to a different resolver.
> [!NOTE]
> This is a reactor setting.
#### DOWNLOADER
Default: `'scrapy.core.downloader.Downloader'`
The downloader to use for crawling.
#### DOWNLOADER_CLIENT_TLS_CIPHERS
Default: `'DEFAULT'`
Use this setting to customize the TLS/SSL ciphers used by the HTTPS download
handler.
The setting should contain a string in the [OpenSSL cipher list format](https://docs.openssl.org/master/man1/openssl-ciphers/#cipher-list-format),
these ciphers will be used as client ciphers. Changing this setting may be
necessary to access certain HTTPS websites: for example, you may need to use
`'DEFAULT:!DH'` for a website with weak DH parameters or enable a
specific cipher that is not included in `DEFAULT` if a website requires it.
Set this setting to `None` to use the default ciphers of the underlying TLS
implementation.
> [!WARNING]
> Changed in version 2.17.0: Added support for setting this to `None`.
> [!NOTE]
> Handling of this setting needs to be implemented inside the download
> handler, so it’s not guaranteed to be supported
> by all 3rd-party handlers.
> [!NOTE]
> ### See also
>
> security-tls-protocols-ciphers
#### DOWNLOAD_TLS_MAX_VERSION
> [!WARNING]
> Added in version 2.17.0.
Default: `None`
Use this setting to change the maximum version of the TLS protocol allowed to
be used by Scrapy.
This setting must be either `None`, in which case it doesn’t affect the
version selection, or one of these string values:
- `'TLSv1.0'`
- `'TLSv1.1'`
- `'TLSv1.2'`
- `'TLSv1.3'`
The range of allowed TLS versions advertised by Scrapy when making TLS
connections will depend on the TLS implementation defaults and the values of
`DOWNLOAD_TLS_MIN_VERSION` and `DOWNLOAD_TLS_MAX_VERSION`.
It’s possible to re-enable versions that are supported by the TLS
implementation but disabled by default by adjusting these settings, but it’s
impossible to enable unsupported ones, such as any versions below 1.2 in many
modern environments.
> [!NOTE]
> Handling of this setting needs to be implemented inside the download
> handler, so it’s not guaranteed to be supported
> by all 3rd-party handlers. Additionally, the set of supported TLS versions
> depends on the TLS implementation being used by the handler.
> [!NOTE]
> ### See also
>
> security-tls-protocols-ciphers
#### DOWNLOAD_TLS_MIN_VERSION
> [!WARNING]
> Added in version 2.17.0.
Default: `None`
Use this setting to change the minimum version of the TLS protocol allowed to
be used by Scrapy.
See `DOWNLOAD_TLS_MAX_VERSION` for the details and limitations.
> [!NOTE]
> ### See also
>
> security-tls-protocols-ciphers
#### DOWNLOADER_CLIENT_TLS_VERBOSE_LOGGING
Default: `False`
Setting this to `True` will enable DEBUG level messages about TLS connection
parameters after establishing HTTPS connections. The kind of information logged
depends on the implementation of the download handler and the versions of
the TLS-related libraries.
> [!NOTE]
> Handling of this setting needs to be implemented inside the download
> handler, so it’s not guaranteed to be supported
> by all 3rd-party handlers.
#### DOWNLOADER_MIDDLEWARES
Default: `{}`
A dict containing the downloader middlewares enabled in your project, and their
orders. For more info see topics-downloader-middleware-setting.
#### DOWNLOADER_MIDDLEWARES_BASE
Default:
```python
{
"scrapy.downloadermiddlewares.offsite.OffsiteMiddleware": 50,
"scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware": 100,
"scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware": 300,
"scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware": 350,
"scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware": 400,
"scrapy.downloadermiddlewares.useragent.UserAgentMiddleware": 500,
"scrapy.downloadermiddlewares.retry.RetryMiddleware": 550,
"scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware": 580,
"scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware": 590,
"scrapy.downloadermiddlewares.redirect.RedirectMiddleware": 600,
"scrapy.downloadermiddlewares.cookies.CookiesMiddleware": 700,
"scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware": 750,
"scrapy.downloadermiddlewares.stats.DownloaderStats": 850,
"scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware": 900,
}
```
A dict containing the downloader middlewares enabled by default in Scrapy. Low
orders are closer to the engine, high orders are closer to the downloader. You
should never modify this setting in your project, modify
`DOWNLOADER_MIDDLEWARES` instead. For more info see
topics-downloader-middleware-setting.
#### DOWNLOADER_STATS
Default: `True`
Whether to enable downloader stats collection.
#### DOWNLOAD_DELAY
Default: `1` (fallback: `0`)
Minimum seconds to wait between 2 consecutive requests to the same domain.
Use `DOWNLOAD_DELAY` to throttle your crawling speed, to avoid hitting
servers too hard.
Decimal numbers are supported. For example, to send a maximum of 4 requests
every 10 seconds:
```python
DOWNLOAD_DELAY = 2.5
```
This setting is also affected by the `DOWNLOAD_DELAY_JITTER` setting,
which randomizes delays by ±50% by default.
Note that `DOWNLOAD_DELAY` can lower the effective per-domain
concurrency below `CONCURRENT_REQUESTS_PER_DOMAIN`. If the response
time of a domain is lower than `DOWNLOAD_DELAY`, the effective
concurrency for that domain is 1. When testing throttling configurations, it
usually makes sense to lower `CONCURRENT_REQUESTS_PER_DOMAIN` first,
and only increase `DOWNLOAD_DELAY` once
`CONCURRENT_REQUESTS_PER_DOMAIN` is 1 but a higher throttling is
desired.
It is possible to change this setting per domain by using
`DOWNLOAD_SLOTS`.
#### DOWNLOAD_DELAY_JITTER
> [!WARNING]
> Added in version 2.19.0.
Default: `0.5`
Magnitude of the random variation applied to `DOWNLOAD_DELAY`, e.g.
`0.2` spreads delays between 80% and 120% of `DOWNLOAD_DELAY`. `0`
disables randomization.
Randomizing delays makes the time between requests less uniform, resulting in a
more natural crawling pattern.
It is possible to change this setting per domain by using
`DOWNLOAD_SLOTS`.
#### DOWNLOAD_BIND_ADDRESS
Default: `None`
The default local outgoing address for download-handler connections.
This setting can be either:
- a host address as a string (e.g. `"127.0.0.2"`), in which case the local
port is chosen automatically, or
- a `(host, port)` tuple (e.g. `("127.0.0.2", 50000)`) to bind to both a
specific local interface and a specific local port.
For example:
```python
# Bind to this local address
DOWNLOAD_BIND_ADDRESS = "127.0.0.2"
```
```python
# Bind to this local address and local port
DOWNLOAD_BIND_ADDRESS = ("127.0.0.2", 5000)
```
If set, built-in HTTP download handlers use this value by default.
Set the `bindaddress` request meta key to override it for a specific
request.
> [!NOTE]
> Handling of this setting needs to be implemented inside the download
> handler, so it’s not guaranteed to be supported
> by all 3rd-party handlers. Specifying the port is unsupported by
> `HttpxDownloadHandler`.
#### DOWNLOAD_HANDLERS
Default: `{}`
A dict containing the download handlers
enabled in your project.
See `DOWNLOAD_HANDLERS_BASE` for example format.
> [!NOTE]
> ### See also
>
> security-unencrypted-protocols and
> security-local-resources
#### DOWNLOAD_HANDLERS_BASE
Default:
```python
{
"data": "scrapy.core.downloader.handlers.datauri.DataURIDownloadHandler",
"file": "scrapy.core.downloader.handlers.file.FileDownloadHandler",
"http": "scrapy.core.downloader.handlers.http11.HTTP11DownloadHandler",
"https": "scrapy.core.downloader.handlers.http11.HTTP11DownloadHandler",
"s3": "scrapy.core.downloader.handlers.s3.S3DownloadHandler",
"ftp": "scrapy.core.downloader.handlers.ftp.FTPDownloadHandler",
}
```
(when `TWISTED_REACTOR_ENABLED` is `True`)
```python
{
"data": "scrapy.core.downloader.handlers.datauri.DataURIDownloadHandler",
"file": "scrapy.core.downloader.handlers.file.FileDownloadHandler",
"http": "scrapy.core.downloader.handlers._aiohttp.AiohttpDownloadHandler",
"https": "scrapy.core.downloader.handlers._aiohttp.AiohttpDownloadHandler",
"s3": "scrapy.core.downloader.handlers.s3.S3DownloadHandler",
"ftp": None,
}
```
(when `TWISTED_REACTOR_ENABLED` is `False`)
A dict containing the download handlers
enabled by default in Scrapy. You should never modify this setting in your
project, modify `DOWNLOAD_HANDLERS` instead.
You can disable any of these download handlers by assigning `None` to their
URI scheme in `DOWNLOAD_HANDLERS`. E.g., to disable the built-in FTP
handler (without replacement), place this in your `settings.py`:
```python
DOWNLOAD_HANDLERS = {
"ftp": None,
}
```
> [!NOTE]
> ### See also
>
> security-unencrypted-protocols and
> security-local-resources
#### DOWNLOAD_SLOTS
Default: `{}`
Allows to define concurrency/delay parameters on per slot (domain) basis:
> ```python
> DOWNLOAD_SLOTS = {
> "quotes.toscrape.com": {"concurrency": 1, "delay": 2, "jitter": 0},
> "books.toscrape.com": {"delay": 3, "jitter": 0.2},
> }
> ```
> [!NOTE]
> For other downloader slots default settings values will be used:
>
> - `DOWNLOAD_DELAY`: `delay`
> - `CONCURRENT_REQUESTS_PER_DOMAIN`: `concurrency`
> - `DOWNLOAD_DELAY_JITTER`: `jitter`
Requests are assigned to a slot based on their URL domain. To assign a request
to a specific slot instead, set the name of the slot as the `download_slot`
`Request.meta` key. Once a request is assigned to a
slot, that key holds the name of the slot.
Since that key is kept on redirects, a redirected request stays in the slot of
the request it comes from, even when it points to a different domain.
#### DOWNLOAD_TIMEOUT
Default: `180`
The amount of time (in secs) that the downloader will wait before timing out.
> [!NOTE]
> This timeout can be per-request using the `download_timeout`
> `Request.meta` key.
> [!NOTE]
> Handling of this setting needs to be implemented inside the download
> handler, so it’s not guaranteed to be supported
> by all 3rd-party handlers.
#### DOWNLOAD_MAXSIZE
Default: `1073741824` (1 GiB)
The maximum response body size (in bytes) allowed. Bigger responses are
aborted and ignored.
This applies both before and after compression. If decompressing a response
body would exceed this limit, decompression is aborted and the response is
ignored.
Use `0` to disable this limit.
> [!NOTE]
> This limit can be set per-request using the `download_maxsize`
> `Request.meta` key.
> [!NOTE]
> Checking responses before decompressing them needs to be implemented inside
> the download handler, so it’s not
> guaranteed to be supported by all 3rd-party handlers.
#### DOWNLOAD_WARNSIZE
Default: `33554432` (32 MiB)
If the size of a response exceeds this value, before or after compression, a
warning will be logged about it.
Use `0` to disable this limit.
> [!NOTE]
> This limit can be set per-request using the `download_warnsize`
> `Request.meta` key.
> [!NOTE]
> Checking responses before decompressing them needs to be implemented inside
> the download handler, so it’s not
> guaranteed to be supported by all 3rd-party handlers.
#### DOWNLOAD_FAIL_ON_DATALOSS
Default: `True`
Whether or not to fail on broken responses, that is, when the declared
`Content-Length` does not match content sent by the server or a chunked
response was not properly finished. If `True`, these responses raise a
`ResponseDataLossError` exception. If `False`, these
responses are passed through and the flag `dataloss` is added to the
response, i.e.: `'dataloss' in response.flags` is `True`.
Optionally, this can be set per-request basis by using the
`download_fail_on_dataloss` Request.meta key to `False`.
> [!NOTE]
> A broken response, or data loss error, may happen under several
> circumstances, from server misconfiguration to network errors to data
> corruption. It is up to the user to decide if it makes sense to process
> broken responses considering they may contain partial or incomplete content.
> If `RETRY_ENABLED` is `True` and this setting is set to `True`,
> the `ResponseDataLossError` failure will be retried
> as usual.
> [!NOTE]
> Handling of this setting needs to be implemented inside the download
> handler, so it’s not guaranteed to be supported
> by all 3rd-party handlers.
> [!WARNING]
> This setting is ignored by the
> `H2DownloadHandler`
> download handler. In case of a data loss
> error, the corresponding HTTP/2 connection may be corrupted, affecting other
> requests that use the same connection; hence, a `ResponseFailed([InvalidBodyLengthError])`
> failure is always raised for every request that was using that connection.
#### DOWNLOAD_VERIFY_CERTIFICATES
Default: `False`
Whether the HTTPS download handlers should verify the server TLS certificate
when making a request and abort the request if the verification fails.
> [!NOTE]
> Handling of this setting needs to be implemented inside the download
> handler, so it’s not guaranteed to be supported
> by all 3rd-party handlers. The exact behavior of a handler (e.g. whether
> certificate problems are logged when this setting is set to `False`)
> depends on its implementation.
> [!NOTE]
> ### See also
>
> security-certificate-verification
#### DUPEFILTER_CLASS
Default: `'scrapy.dupefilters.RFPDupeFilter'`
The class used to detect and filter duplicate requests.
The default, `RFPDupeFilter`, filters based on the
`REQUEST_FINGERPRINTER_CLASS` setting.
To change how duplicates are checked, you can point `DUPEFILTER_CLASS`
to a custom subclass of `RFPDupeFilter` that
overrides its `__init__` method to use a different request
fingerprinting class. For example:
```python
from scrapy.dupefilters import RFPDupeFilter
from scrapy.utils.request import fingerprint
class CustomRequestFingerprinter:
def fingerprint(self, request):
return fingerprint(request, include_headers=["X-ID"])
class CustomDupeFilter(RFPDupeFilter):
def __init__(self, path=None, debug=False, *, fingerprinter=None):
super().__init__(
path=path, debug=debug, fingerprinter=CustomRequestFingerprinter()
)
```
To disable duplicate request filtering set `DUPEFILTER_CLASS` to
`'scrapy.dupefilters.BaseDupeFilter'`. Note that not filtering out duplicate
requests may cause crawling loops. It is usually better to set
the `dont_filter` parameter to `True` on the `__init__` method of a
specific `Request` object that should not be filtered out.
A class assigned to `DUPEFILTER_CLASS` must implement the following
interface:
```python
class MyDupeFilter:
@classmethod
def from_crawler(cls, crawler):
"""Returns an instance of this duplicate request filtering class
based on the current Crawler instance."""
return cls()
def request_seen(self, request):
"""Returns ``True`` if *request* is a duplicate of another request
seen in a previous call to :meth:`request_seen`, or ``False``
otherwise."""
return False
def open(self):
"""Called before the spider opens. It may return a deferred."""
pass
def close(self, reason):
"""Called before the spider closes. It may return a deferred."""
pass
def log(self, request, spider):
"""Logs that a request has been filtered out.
It is called right after a call to :meth:`request_seen` that
returns ``True``.
If :meth:`request_seen` always returns ``False``, such as in the
case of :class:`~scrapy.dupefilters.BaseDupeFilter`, this method
may be omitted.
"""
pass
```
#### *class* scrapy.dupefilters.BaseDupeFilter
Dummy duplicate request filtering class (`DUPEFILTER_CLASS`)
that does not filter out any request.
#### *class* scrapy.dupefilters.RFPDupeFilter(path: str | None = None, debug: bool = False, \*, fingerprinter: RequestFingerprinterProtocol | None = None)
Duplicate request filtering class (`DUPEFILTER_CLASS`) that
filters out requests with the canonical
(`w3lib.url.canonicalize_url()`) `url`,
`method` and `body`.
##### Job directory contents
> [!WARNING]
> The files that this class generates in the job directory are an implementation detail, and may change without a
> warning in a future version of Scrapy. Do not rely on the following
> information for anything other than debugging purposes.
When using `JOBDIR`, seen fingerprints are tracked in a binary
file named `requests.seen` in the job directory,
where each fingerprint is stored as its big-endian, 2-byte length followed
by the fingerprint itself.
#### DUPEFILTER_DEBUG
Default: `False`
By default, `RFPDupeFilter` only logs the first duplicate request.
Setting `DUPEFILTER_DEBUG` to `True` will make it log all duplicate requests.
#### EDITOR
Default: `vi` (on Unix systems) or the IDLE editor (on Windows)
The editor to use for editing spiders with the `edit` command.
Additionally, if the `EDITOR` environment variable is set, the `edit`
command will prefer it over the default setting.
#### EXTENSIONS
Default: `{}`
Component priority dictionary of
enabled extensions. See topics-extensions.
#### EXTENSIONS_BASE
Default:
```python
{
"scrapy.extensions.corestats.CoreStats": 0,
"scrapy.extensions.logcount.LogCount": 0,
"scrapy.extensions.telnet.TelnetConsole": 0,
"scrapy.extensions.memusage.MemoryUsage": 0,
"scrapy.extensions.memdebug.MemoryDebugger": 0,
"scrapy.extensions.closespider.CloseSpider": 0,
"scrapy.extensions.feedexport.FeedExporter": 0,
"scrapy.extensions.logstats.LogStats": 0,
"scrapy.extensions.spiderstate.SpiderState": 0,
"scrapy.extensions.throttle.AutoThrottle": 0,
"scrapy.extensions.remote_control.RemoteControl": 0,
}
```
A dict containing the extensions available by default in Scrapy, and their
orders. This setting contains all stable built-in extensions. Keep in mind that
some of them need to be enabled through a setting.
For more information See the extensions user guide
and the list of available extensions.
#### FEED_TEMPDIR
Default: `None`
The Feed Temp dir allows you to set a custom folder to save crawler
temporary files before uploading with FTP feed storage and
Amazon S3.
#### FEED_STORAGE_GCS_ACL
Default: `""`
The Access Control List (ACL) used when storing items to Google Cloud Storage.
For more information on how to set this value, please refer to the column *JSON API* in [Google Cloud documentation](https://docs.cloud.google.com/storage/docs/access-control/lists).
#### FORCE_CRAWLER_PROCESS
Default: `False`
If `False`, Scrapy commands that need a CrawlerProcess, when `TWISTED_REACTOR_ENABLED`
is set to `True`, will decide between using
`scrapy.crawler.AsyncCrawlerProcess` and
`scrapy.crawler.CrawlerProcess` based on the value of the
`TWISTED_REACTOR` setting, but ignoring its value in per-spider
settings.
If `True`, these commands will always use
`CrawlerProcess` when `TWISTED_REACTOR_ENABLED`
is set to `True`.
When `TWISTED_REACTOR_ENABLED` is set to `False`,
`AsyncCrawlerProcess` will be used in all cases.
Set this to `True` if you want to set `TWISTED_REACTOR` to a
non-default value in per-spider settings.
> [!NOTE]
> This is a pre-crawler setting.
#### FTP_PASSIVE_MODE
Default: `True`
Whether or not to use passive mode when initiating FTP transfers, unless there
is an `"ftp_passive"` key in `Request` meta.
> [!NOTE]
> Handling of this setting needs to be implemented inside the download
> handler, so it’s not guaranteed to be supported
> by all 3rd-party handlers.
#### FTP_PASSWORD
Default: `"guest"`
The password to use for FTP connections when there is no `"ftp_password"`
in `Request` meta.
> [!NOTE]
> Paraphrasing [RFC 1635](https://datatracker.ietf.org/doc/html/rfc1635), although it is common to use either the password
> “guest” or one’s e-mail address for anonymous FTP,
> some FTP servers explicitly ask for the user’s e-mail address
> and will not allow login with the “guest” password.
> [!NOTE]
> Handling of this setting needs to be implemented inside the download
> handler, so it’s not guaranteed to be supported
> by all 3rd-party handlers.
#### FTP_USER
Default: `"anonymous"`
The username to use for FTP connections when there is no `"ftp_user"`
in `Request` meta.
> [!NOTE]
> Handling of this setting needs to be implemented inside the download
> handler, so it’s not guaranteed to be supported
> by all 3rd-party handlers.
#### GCS_PROJECT_ID
Default: `None`
The Project ID that will be used when storing data on [Google Cloud Storage](https://cloud.google.com/storage/).
#### HTTP2_MAX_FRAME_SIZE
> [!WARNING]
> Added in version 2.18.0.
Default: `16384`
Maximum [frame size](https://datatracker.ietf.org/doc/html/rfc7540#section-4.2), in bytes, that servers may send, between `16384` and
`16777215`. Connections to servers that send a larger frame fail.
Raise it for servers that send larger frames regardless of this value. Note
that `DOWNLOAD_MAXSIZE` and `DOWNLOAD_WARNSIZE` are checked
once per received frame, so a higher value allows a response to exceed them by
more before being caught.
`HttpxDownloadHandler` ignores
this setting, as `httpx` does not allow configuring the frame size.
#### ITEM_PIPELINES
Default: `{}`
A dict containing the item pipelines to use, and their orders. Order values are
arbitrary, but it is customary to define them in the 0-1000 range. Lower orders
process before higher orders.
Example:
```python
ITEM_PIPELINES = {
"mybot.pipelines.validate.ValidateMyItem": 300,
"mybot.pipelines.validate.StoreMyItem": 800,
}
```
#### ITEM_PIPELINES_BASE
Default: `{}`
A dict containing the pipelines enabled by default in Scrapy. You should never
modify this setting in your project, modify `ITEM_PIPELINES` instead.
#### ITEM_PROCESSOR
Default: `"scrapy.pipelines.ItemPipelineManager"`
The component that builds the item pipeline from `ITEM_PIPELINES` and runs scraped items
through it. It must implement `ItemProcessorProtocol`.
#### *class* scrapy.pipelines.ItemProcessorProtocol(\*args, \*\*kwargs)
Protocol for item processor implementations.
See `ITEM_PROCESSOR`.
##### *async* open_spider_async() → None
Get the item processor ready to process items.
##### *async* process_item_async(item: Any) → Any
Return the processed *item*, or raise
`DropItem` to drop it.
##### *async* close_spider_async() → None
Release any resource that the item processor is using.
#### JOBDIR
Default: `None`
A string indicating the directory for storing the state of a crawl when
pausing and resuming crawls.
#### LOG_COLOR
> [!WARNING]
> Added in version 2.19.0.
Default: `True`
Whether to colorize log output by log level when logging to a terminal.
Requires the `color` extra:
```bash
pip install scrapy[color]
```
> [!NOTE]
> This is a logging setting.
#### LOG_ENABLED
Default: `True`
Whether to enable logging.
> [!NOTE]
> This is a logging setting.
#### LOG_ENCODING
Default: `'utf-8'`
The encoding to use for logging.
> [!NOTE]
> This is a logging setting.
#### LOG_FILE
Default: `None`
File name to use for logging output. If `None`, standard error will be used.
> [!NOTE]
> This is a logging setting.
#### LOG_FILE_APPEND
Default: `True`
If `False`, the log file specified with `LOG_FILE` will be
overwritten (discarding the output from previous runs, if any).
> [!NOTE]
> This is a logging setting.
#### LOG_FORMAT
Default: `'%(asctime)s [%(name)s] %(levelname)s: %(message)s'`
String for formatting log messages. Refer to the
Python logging documentation for the whole
list of available placeholders.
> [!NOTE]
> This is a logging setting.
#### LOG_DATEFORMAT
Default: `'%Y-%m-%d %H:%M:%S'`
String for formatting date/time, expansion of the `%(asctime)s` placeholder
in `LOG_FORMAT`. Refer to the
Python datetime documentation for the
whole list of available directives.
> [!NOTE]
> This is a logging setting.
#### LOG_FORMATTER
Default: `scrapy.logformatter.LogFormatter`
The class to use for formatting log messages for different actions.
#### LOG_INSTALL_ROOT_HANDLER
> [!WARNING]
> Added in version 2.19.0.
Default: `True`
Whether to install a handler for the root logger, configured according to the
other logging settings. Set this to `False` to
manage the root logger yourself, e.g. from a custom command or from a
`settings.py` module executed before Scrapy configures logging.
> [!NOTE]
> This is a logging setting.
#### LOG_LEVEL
Default: `'DEBUG'`
Minimum level to log. Available levels are: CRITICAL, ERROR, WARNING,
INFO, DEBUG. For more info see topics-logging.
> [!NOTE]
> This is a logging setting.
#### LOG_STDOUT
Default: `False`
If `True`, all standard output (and error) of your process will be redirected
to the log. For example if you `print('hello')` it will appear in the Scrapy
log.
> [!NOTE]
> This is a logging setting.
#### LOG_SHORT_NAMES
Default: `False`
If `True`, the logs will just contain the root path. If it is set to `False`
then it displays the component responsible for the log output
> [!NOTE]
> This is a logging setting.
#### LOG_VERSIONS
Default: `["lxml", "libxml2", "cssselect", "parsel", "w3lib", "Twisted", "Python", "pyOpenSSL", "cryptography", "Platform"]`
Logs the installed versions of the specified items.
An item can be any installed Python package.
The following special items are also supported:
- `libxml2`
- `Platform` (`platform.platform()`)
- `Python`
- `pyOpenSSL`
#### LOGSTATS_INTERVAL
Default: `60.0`
The interval (in seconds) between each logging printout of the stats
by `LogStats`.
#### MEMDEBUG_ENABLED
Default: `False`
Whether to enable memory debugging.
#### MEMUSAGE_ENABLED
Default: `True`
Scope: `scrapy.extensions.memusage.MemoryUsage`
Whether to enable the memory usage extension. This extension keeps track of
a peak memory used by the process (it writes it to stats). It can also
optionally shutdown the Scrapy process when it exceeds a memory limit
(see `MEMUSAGE_LIMIT_MB`).
See topics-extensions-ref-memusage.
#### MEMUSAGE_LIMIT_MB
Default: `0`
Scope: `scrapy.extensions.memusage.MemoryUsage`
The maximum amount of memory to allow (in megabytes) before shutting down
Scrapy (if `MEMUSAGE_ENABLED` is `True`). If zero, no check will be
performed.
See topics-extensions-ref-memusage.
#### MEMUSAGE_CHECK_INTERVAL_SECONDS
Default: `60.0`
Scope: `scrapy.extensions.memusage.MemoryUsage`
The Memory usage extension
checks the current memory usage, versus the limits set by
`MEMUSAGE_LIMIT_MB` and `MEMUSAGE_WARNING_MB`,
at fixed time intervals.
This sets the length of these intervals, in seconds.
See topics-extensions-ref-memusage.
#### MEMUSAGE_WARNING_MB
Default: `0`
Scope: `scrapy.extensions.memusage.MemoryUsage`
The maximum amount of memory to allow (in megabytes) before sending a
`memusage_warning_reached` signal (if `MEMUSAGE_ENABLED` is
`True`). If zero, no signal will be sent.
See topics-extensions-ref-memusage.
#### NEWSPIDER_MODULE
Default: `"<project name>.spiders"` (fallback: `""`)
Module where to create new spiders using the `genspider` command.
Example:
```python
NEWSPIDER_MODULE = "mybot.spiders_dev"
```
#### REACTOR_THREADPOOL_MAXSIZE
Default: `10`
The maximum limit for Twisted Reactor thread pool size. This is common
multi-purpose thread pool used by various Scrapy components. Threaded
DNS Resolver, BlockingFeedStorage, S3FilesStore just to name a few. Increase
this value if you’re experiencing problems with insufficient blocking IO.
> [!NOTE]
> This is a reactor setting.
#### REDIRECT_PRIORITY_ADJUST
Default: `+2`
Scope: `scrapy.downloadermiddlewares.redirect.RedirectMiddleware`
Adjust redirect request priority relative to original request:
- **a positive priority adjust (default) means higher priority.**
- a negative priority adjust means lower priority.
#### ROBOTSTXT_OBEY
Default: `True` (fallback: `False`)
If enabled, Scrapy will respect robots.txt policies. For more information see
topics-dlmw-robots.
> [!NOTE]
> While the default value is `False` for historical reasons,
> this option is enabled by default in settings.py file generated
> by `scrapy startproject` command.
#### ROBOTSTXT_PARSER
Default: `'scrapy.robotstxt.ProtegoRobotParser'`
The parser backend to use for parsing `robots.txt` files. For more information see
topics-dlmw-robots.
#### ROBOTSTXT_USER_AGENT
Default: `None`
The user agent string to use for matching in the robots.txt file. If `None`,
the User-Agent header you are sending with the request or the
`USER_AGENT` setting (in that order) will be used for determining
the user agent to use in the robots.txt file.
#### SCHEDULER
Default: `Scheduler`
The scheduler class to be used for crawling. See topics-scheduler for
details.
#### SCHEDULER_DEBUG
Default: `False`
Setting to `True` will log debug information about the requests scheduler.
This currently logs (only once) if the requests cannot be serialized to disk.
The `scheduler/unserializable` stat tracks the number of times this
happens.
Example entry in logs:
```
1956-01-31 00:00:00+0800 [scrapy.core.scheduler] ERROR: Unable to serialize request:
<GET http://example.com> - reason: cannot serialize <Request at 0x9a7c7ec>
(type Request)> - no more unserializable requests will be logged
(see 'scheduler/unserializable' stats counter)
```
#### SCHEDULER_DISK_QUEUE
Default: `'scrapy.squeues.PickleLifoDiskQueue'`
> [!WARNING]
> Added in version 2.19.0: The `SQLite` queue types.
Type of disk queue that will be used by the scheduler. Other available types
are `scrapy.squeues.PickleFifoDiskQueue`,
`scrapy.squeues.MarshalFifoDiskQueue`,
`scrapy.squeues.MarshalLifoDiskQueue`,
`scrapy.squeues.PickleFifoSQLiteQueue`,
`scrapy.squeues.PickleLifoSQLiteQueue`,
`scrapy.squeues.MarshalFifoSQLiteQueue` and
`scrapy.squeues.MarshalLifoSQLiteQueue`.
The `SQLite` types store requests in an SQLite database, which makes writes
slower but keeps the queue usable after an unclean shutdown. See
security-job-state.
#### SCHEDULER_MEMORY_QUEUE
Default: `'scrapy.squeues.LifoMemoryQueue'`
Type of in-memory queue used by the scheduler. Other available type is:
`scrapy.squeues.FifoMemoryQueue`.
#### SCHEDULER_PRIORITY_QUEUE
Default: `DownloaderAwarePriorityQueue`
Type of priority queue used by the scheduler.
Another available type is `ScrapyPriorityQueue`.
`DownloaderAwarePriorityQueue` works better than
`ScrapyPriorityQueue` when you crawl many different
domains in parallel.
#### SCHEDULER_START_DISK_QUEUE
Default: `'scrapy.squeues.PickleFifoDiskQueue'`
Type of disk queue (see `JOBDIR`) that the scheduler uses for start requests.
For available choices, see `SCHEDULER_DISK_QUEUE`.
Use `None` or `""` to disable these separate queues entirely, and instead
have start requests share the same queues as other requests.
> [!NOTE]
> Disabling separate start request queues makes start request order unintuitive: start requests will be sent in order
> only until `CONCURRENT_REQUESTS` is reached, then remaining start
> requests will be sent in reverse order.
#### SCHEDULER_START_MEMORY_QUEUE
Default: `'scrapy.squeues.FifoMemoryQueue'`
Type of in-memory queue that the scheduler uses for
start requests.
For available choices, see `SCHEDULER_MEMORY_QUEUE`.
Use `None` or `""` to disable these separate queues entirely, and instead
have start requests share the same queues as other requests.
> [!NOTE]
> Disabling separate start request queues makes start request order unintuitive: start requests will be sent in order
> only until `CONCURRENT_REQUESTS` is reached, then remaining start
> requests will be sent in reverse order.
#### SCRAPER_SLOT_MAX_ACTIVE_SIZE
Default: `5_000_000`
Soft limit (in bytes) for response data being processed.
While the sum of the sizes of all responses being processed is above this value,
Scrapy does not process new requests.
#### SPIDER_CONTRACTS
Default: `{}`
A dict containing the spider contracts enabled in your project, used for
testing spiders. For more info see topics-contracts.
#### SPIDER_CONTRACTS_BASE
Default:
```python
{
"scrapy.contracts.default.UrlContract": 1,
"scrapy.contracts.default.CallbackKeywordArgumentsContract": 1,
"scrapy.contracts.default.MetadataContract": 1,
"scrapy.contracts.default.ReturnsContract": 2,
"scrapy.contracts.default.ScrapesContract": 3,
}
```
A dict containing the Scrapy contracts enabled by default in Scrapy. You should
never modify this setting in your project, modify `SPIDER_CONTRACTS`
instead. For more info see topics-contracts.
You can disable any of these contracts by assigning `None` to their class
path in `SPIDER_CONTRACTS`. E.g., to disable the built-in
`ScrapesContract`, place this in your `settings.py`:
```python
SPIDER_CONTRACTS = {
"scrapy.contracts.default.ScrapesContract": None,
}
```
#### SPIDER_LOADER_CLASS
Default: `'scrapy.spiderloader.SpiderLoader'`
The class that will be used for loading spiders, which must implement the
topics-api-spiderloader.
> [!NOTE]
> This is a pre-crawler setting.
#### SPIDER_LOADER_WARN_ONLY
Default: `False`
By default, when Scrapy tries to import spider classes from `SPIDER_MODULES`,
it will fail loudly if there is any `ImportError` or `SyntaxError` exception.
But you can choose to silence this exception and turn it into a simple
warning by setting `SPIDER_LOADER_WARN_ONLY = True`.
> [!NOTE]
> This is a pre-crawler setting.
#### SPIDER_MIDDLEWARES
Default: `{}`
A dict containing the spider middlewares enabled in your project, and their
orders. For more info see topics-spider-middleware-setting.
#### SPIDER_MIDDLEWARES_BASE
Default:
```python
{
"scrapy.spidermiddlewares.start.StartSpiderMiddleware": 25,
"scrapy.spidermiddlewares.httperror.HttpErrorMiddleware": 50,
"scrapy.spidermiddlewares.referer.RefererMiddleware": 700,
"scrapy.spidermiddlewares.urllength.UrlLengthMiddleware": 800,
"scrapy.spidermiddlewares.depth.DepthMiddleware": 900,
}
```
A dict containing the spider middlewares enabled by default in Scrapy, and
their orders. Low orders are closer to the engine, high orders are closer to
the spider. For more info see topics-spider-middleware-setting.
#### SPIDER_MODULES
Default: `["<project name>.spiders"]` (fallback: `[]`)
A list of modules where Scrapy will look for spiders.
Example:
```python
SPIDER_MODULES = ["mybot.spiders_prod", "mybot.spiders_dev"]
```
> [!NOTE]
> This is a pre-crawler setting.
#### STATS_CLASS
Default: `'scrapy.statscollectors.MemoryStatsCollector'`
The class to use for collecting stats, who must implement the
topics-api-stats.
#### STATS_DUMP
Default: `True`
Dump the Scrapy stats (to the Scrapy log) once the spider
finishes.
For more info see: topics-stats.
#### TELNETCONSOLE_ENABLED
Default: `True` (`False` when `TWISTED_REACTOR_ENABLED` is `False`)
A boolean which specifies if the telnet console
will be enabled (provided its extension is also enabled).
> [!NOTE]
> ### See also
>
> security-telnet
#### TEMPLATES_DIR
Default: `templates` dir inside scrapy module
The directory where to look for templates when creating new projects with
`startproject` command and new spiders with `genspider`
command. See spider-templates.
The project name must not conflict with the name of custom files or directories
in the `project` subdirectory.
#### TWISTED_DNS_RESOLVER
Default: `'scrapy.resolver.CachingThreadedResolver'`
The class to be used by Twisted to resolve DNS names. The default
`scrapy.resolver.CachingThreadedResolver` supports specifying a timeout for
DNS requests via the `DNS_TIMEOUT` setting, but works only with IPv4
addresses. Scrapy provides an alternative resolver,
`scrapy.resolver.CachingHostnameResolver`, which supports IPv4/IPv6 addresses but does not
take the `DNS_TIMEOUT` setting into account.
> [!NOTE]
> This setting has no effect when `TWISTED_REACTOR_ENABLED` is `False`.
> [!NOTE]
> This is a reactor setting.
#### TWISTED_REACTOR_ENABLED
Default: `True`
Whether to install and use the Twisted reactor.
If this is set to `True`, Scrapy will use the Twisted reactor and will
install one according to the `TWISTED_REACTOR` setting value when
appropriate (e.g. when running via the command-line tool). This is the traditional mode of using Scrapy.
If this is set to `False`, Scrapy will use the asyncio event loop directly
and will not attempt to install or use a reactor. Features that require a
reactor won’t be available, but Twisted APIs that don’t require a reactor,
including `Deferred` and
`Failure`, will still be available. On the other
hand, limitations related to Twisted reactors (such as not being able to start
a reactor in the same process where a reactor was previously started and
stopped) will not apply. This mode is currently experimental and may not be
suitable for production use. It may also not be supported by 3rd-party code.
See asyncio-without-reactor for more information about this mode.
> [!NOTE]
> This is a pre-crawler setting.
> [!WARNING]
> Added in version 2.15.0.
#### TWISTED_REACTOR
Default: `"twisted.internet.asyncioreactor.AsyncioSelectorReactor"`
Import path of a given `reactor`.
> [!NOTE]
> This setting has no effect when `TWISTED_REACTOR_ENABLED` is `False`.
Scrapy will install this reactor if no other reactor is installed yet, such as
when the `scrapy` CLI program is invoked or when using the
`AsyncCrawlerProcess` class or the
`CrawlerProcess` class.
If you are using the `AsyncCrawlerRunner` class or the
`CrawlerRunner` class, you also
need to install the correct reactor manually. You can do that using
`install_reactor()`:
#### scrapy.utils.reactor.install_reactor(reactor_path: str, event_loop_path: str | None = None) → None
Installs the `reactor` with the specified
import path. Also installs the asyncio event loop with the specified import
path if the asyncio reactor is enabled
If a reactor is already installed,
`install_reactor()` has no effect.
`AsyncCrawlerRunner` and other similar classes raise an
exception if the installed reactor does not match the
`TWISTED_REACTOR` setting; therefore, having top-level
`reactor` imports in project files and imported
third-party libraries will make Scrapy raise an exception when it checks which
reactor is installed.
In order to use the reactor installed by Scrapy:
```python
import scrapy
from twisted.internet import reactor
class QuotesSpider(scrapy.Spider):
name = "quotes"
def __init__(self, *args, **kwargs):
self.timeout = int(kwargs.pop("timeout", "60"))
super().__init__(*args, **kwargs)
async def start(self):
reactor.callLater(self.timeout, self.stop)
urls = ["https://quotes.toscrape.com/page/1"]
for url in urls:
yield scrapy.Request(url=url, callback=self.parse)
def parse(self, response):
for quote in response.css("div.quote"):
yield {"text": quote.css("span.text::text").get()}
def stop(self):
self.crawler.engine.close_spider(self, "timeout")
```
which raises an exception, becomes:
```python
import scrapy
class QuotesSpider(scrapy.Spider):
name = "quotes"
def __init__(self, *args, **kwargs):
self.timeout = int(kwargs.pop("timeout", "60"))
super().__init__(*args, **kwargs)
async def start(self):
from twisted.internet import reactor
reactor.callLater(self.timeout, self.stop)
urls = ["https://quotes.toscrape.com/page/1"]
for url in urls:
yield scrapy.Request(url=url, callback=self.parse)
def parse(self, response):
for quote in response.css("div.quote"):
yield {"text": quote.css("span.text::text").get()}
def stop(self):
self.crawler.engine.close_spider(self, "timeout")
```
If this setting is set `None`, Scrapy will use the existing reactor if one is
already installed, or install the default reactor defined by Twisted for the
current platform.
> [!WARNING]
> Changed in version 2.13: The default value was changed from `None` to
> `"twisted.internet.asyncioreactor.AsyncioSelectorReactor"`.
For additional information, see core/howto/choosing-reactor.
> [!NOTE]
> This is a reactor setting.
#### URLLENGTH_LIMIT
Default: `2083`
Scope: `scrapy.spidermiddlewares.urllength`
The maximum URL length to allow for crawled URLs.
This setting can act as a stopping condition in case of URLs of ever-increasing
length, which may be caused for example by a programming error either in the
target server or in your code. See also `REDIRECT_MAX_TIMES` and
`DEPTH_LIMIT`.
Use `0` to allow URLs of any length.
The default value is copied from the [Microsoft Internet Explorer maximum URL
length](https://web.archive.org/web/20250206050143/https://support.microsoft.com/en-us/topic/maximum-url-length-is-2-083-characters-in-internet-explorer-174e7c8a-6666-f4e0-6fd6-908b53c12246), even though this setting exists for different reasons.
#### USER_AGENT
Default: `"Scrapy/VERSION (+https://scrapy.org)"`
The default User-Agent to use when crawling, unless overridden. This user agent is
also used by `RobotsTxtMiddleware`
if `ROBOTSTXT_USER_AGENT` setting is `None` and
there is no overriding User-Agent header specified for the request.
Set it to a value that identifies you, including a URL or an email address
where website owners can reach you, e.g. `"MyProject
(+https://example.com/bot)"`, so that they can ask you to adjust your crawler
rather than block it.
#### WARN_ON_GENERATOR_RETURN_VALUE
Default: `True`
When enabled, Scrapy will warn if generator-based callback methods (like
`parse`) contain return statements with non-`None` values. This helps detect
potential mistakes in spider development.
Disable this setting to prevent syntax errors that may occur when dynamically
modifying generator function source code during runtime, skip AST parsing of
callback functions, or improve performance in auto-reloading development
environments.
Source: /topics/exceptions.md
## Exceptions
Here’s a list of all exceptions included in Scrapy and their usage, except for
the download handler exceptions.
#### *exception* scrapy.exceptions.CloseSpider(reason: str = 'cancelled')
Raised from a spider callback, or while the
spider is starting, to request the spider to be closed/stopped.
> [!WARNING]
> Changed in version 2.18.0: Added support for raising it while the spider is starting.
*reason* is a string with the reason for closing.
For example:
```python
def parse_page(self, response):
if "Bandwidth exceeded" in response.text:
raise CloseSpider("bandwidth_exceeded")
```
#### *exception* scrapy.exceptions.DontCloseSpider
Raised in a `spider_idle` signal handler to prevent the spider
from being closed.
#### *exception* scrapy.exceptions.DropItem(message: str, log_level: str | None = None)
Raised from the `process_item()` method of an item pipeline to stop the processing of an item.
#### *exception* scrapy.exceptions.IgnoreRequest
Raised to indicate that a request should be ignored.
A downloader middleware can raise it
from its
`process_request()`
or
`process_response()`
method to drop a request, and a `request_scheduled` signal handler
can raise it to drop a request before it reaches the
scheduler.
#### *exception* scrapy.exceptions.NotConfigured
Raised by a component from its `__init__()`
or `from_crawler()` method to indicate that it will remain disabled.
Only the following components can be disabled this way:
- Downloader middlewares
- Extensions
- Item pipelines
- Spider middlewares
#### *exception* scrapy.exceptions.NotSupported
Raised to indicate that a requested feature is not supported.
For example, Scrapy raises it when text-parsing shortcuts such as
`response.css()` or
`response.xpath()` are used on a
`Response` whose content is not text, or when sending a
request whose URL scheme has no matching download handler.
#### *exception* scrapy.exceptions.StopDownload(\*, fail: bool = True)
Raised from a `bytes_received` or
`headers_received` signal handler to stop the
download of the response body.
The `fail` boolean parameter controls which method will handle the
resulting response:
* If `fail=True` (default), the request errback is called. The response
object is available as the `response` attribute of the `StopDownload`
exception, which is in turn stored as the `value` attribute of the
received `Failure` object. This means that
in an errback defined as `def errback(self, failure)`, the response can
be accessed though `failure.value.response`.
* If `fail=False`, the request callback is called instead.
In both cases, the response could have its body truncated: the body contains
all bytes received up until the exception is raised, including the bytes
received in the signal handler that raises the exception. Also, the response
object is marked with `"download_stopped"` in its
`flags` attribute.
Source: /topics/logging.md
## Logging
Scrapy uses `logging` for event logging. We’ll
provide some simple examples to get you started, but for more advanced
use-cases it’s strongly suggested to read thoroughly its documentation.
Logging works out of the box, and can be configured to some extent with the
Scrapy settings listed in topics-logging-settings.
Scrapy calls `scrapy.utils.log.configure_logging()` to set some reasonable
defaults and handle those settings in topics-logging-settings when
running commands, so it’s recommended to manually call it if you’re running
Scrapy from scripts as described in run-from-script.
### Log levels
Python’s builtin logging defines 5 different levels to indicate the severity of a
given log message. Here are the standard ones, listed in decreasing order:
1. `logging.CRITICAL` - for critical errors (highest severity)
2. `logging.ERROR` - for regular errors
3. `logging.WARNING` - for warning messages
4. `logging.INFO` - for informational messages
5. `logging.DEBUG` - for debugging messages (lowest severity)
### How to log messages
Here’s a quick example of how to log a message using the `logging.WARNING`
level:
```python
import logging
logging.warning("This is a warning")
```
There are shortcuts for issuing log messages on any of the standard 5 levels,
and there’s also a general `logging.log` method which takes a given level as
argument. If needed, the last example could be rewritten as:
```python
import logging
logging.log(logging.WARNING, "This is a warning")
```
On top of that, you can create different “loggers” to encapsulate messages. (For
example, a common practice is to create different loggers for every module).
These loggers can be configured independently, and they allow hierarchical
constructions.
The previous examples use the root logger behind the scenes, which is a top level
logger where all messages are propagated to (unless otherwise specified). Using
`logging` helpers is merely a shortcut for getting the root logger
explicitly, so this is also an equivalent of the last snippets:
```python
import logging
logger = logging.getLogger()
logger.warning("This is a warning")
```
You can use a different logger just by getting its name with the
`logging.getLogger` function:
```python
import logging
logger = logging.getLogger("mycustomlogger")
logger.warning("This is a warning")
```
Finally, you can ensure having a custom logger for any module you’re working on
by using the `__name__` variable, which is populated with current module’s
path:
```python
import logging
logger = logging.getLogger(__name__)
logger.warning("This is a warning")
```
> [!NOTE]
> ### See also
>
> Module logging, HowTo
> : Basic Logging Tutorial
>
> Module logging, Loggers
> : Further documentation on loggers
### Logging from Spiders
Scrapy provides a `logger` within each Spider
instance, which can be accessed and used like this:
```python
import scrapy
class MySpider(scrapy.Spider):
name = "myspider"
start_urls = ["https://scrapy.org"]
def parse(self, response):
self.logger.info("Parse function called on %s", response.url)
```
That logger is created using the Spider’s name, but you can use any custom
Python logger you want. For example:
```python
import logging
import scrapy
logger = logging.getLogger("mycustomlogger")
class MySpider(scrapy.Spider):
name = "myspider"
start_urls = ["https://scrapy.org"]
def parse(self, response):
logger.info("Parse function called on %s", response.url)
```
### Logging configuration
Loggers on their own don’t manage how messages sent through them are displayed.
For this task, different “handlers” can be attached to any logger instance and
they will redirect those messages to appropriate destinations, such as the
standard output, files, emails, etc.
By default, Scrapy sets and configures a handler for the root logger, based on
the settings below.
#### Logging settings
These settings can be used to configure the logging:
* `LOG_FILE`
* `LOG_FILE_APPEND`
* `LOG_COLOR`
* `LOG_ENABLED`
* `LOG_ENCODING`
* `LOG_LEVEL`
* `LOG_FORMAT`
* `LOG_DATEFORMAT`
* `LOG_STDOUT`
* `LOG_SHORT_NAMES`
* `LOG_INSTALL_ROOT_HANDLER`
The first couple of settings define a destination for log messages. If
`LOG_FILE` is set, messages sent through the root logger will be
redirected to a file named `LOG_FILE` with encoding
`LOG_ENCODING`. If unset and `LOG_ENABLED` is `True`, log
messages will be displayed on the standard error. If `LOG_FILE` is set
and `LOG_FILE_APPEND` is `False`, the file will be overwritten
(discarding the output from previous runs, if any). Lastly, if
`LOG_ENABLED` is `False`, there won’t be any visible log output.
Set `LOG_INSTALL_ROOT_HANDLER` to `False` to prevent Scrapy from
installing a root logging handler altogether, e.g. to configure the root
logger yourself.
`LOG_LEVEL` determines the minimum level of severity to display, those
messages with lower severity will be filtered out. It ranges through the
possible levels listed in topics-logging-levels.
`LOG_FORMAT` and `LOG_DATEFORMAT` specify formatting strings
used as layouts for all messages. Those strings can contain any placeholders
listed in logging’s logrecord attributes docs and
datetime’s strftime and strptime directives
respectively.
If `LOG_SHORT_NAMES` is set, then the logs will not display the Scrapy
component that prints the log. It is unset by default, hence logs contain the
Scrapy component responsible for that log output.
When logging to a terminal, `LOG_COLOR` colorizes messages by log
level.
#### Rotating log files
Scrapy’s `LOG_FILE` setting writes logs to a single file. It does not
rotate log files automatically, but you can use Python’s standard
`logging.handlers` module when running Scrapy from a script.
For example, to rotate the log file every day:
```python
import logging
from logging.handlers import TimedRotatingFileHandler
from scrapy.crawler import CrawlerProcess
from scrapy.utils.project import get_project_settings
from myproject.spiders.myspider import MySpider
settings = get_project_settings()
settings.set("LOG_INSTALL_ROOT_HANDLER", False)
process = CrawlerProcess(settings)
handler = TimedRotatingFileHandler(
"scrapy.log",
when="midnight",
backupCount=7,
encoding=settings.get("LOG_ENCODING"),
)
handler.setFormatter(
logging.Formatter(settings.get("LOG_FORMAT"), settings.get("LOG_DATEFORMAT"))
)
root_logger = logging.getLogger()
root_logger.setLevel(settings.get("LOG_LEVEL"))
root_logger.addHandler(handler)
process.crawl(MySpider)
process.start()
```
#### Command-line options
There are command-line arguments, available for all commands, that you can use
to override some of the Scrapy settings regarding logging.
* `--logfile FILE`
: Overrides `LOG_FILE`
* `--loglevel/-L LEVEL`
: Overrides `LOG_LEVEL`
* `--nolog`
: Sets `LOG_ENABLED` to `False`
> [!NOTE]
> ### See also
>
> Module `logging.handlers`
> : Further documentation on available handlers
#### Custom Log Formats
A custom log format can be set for different actions by extending
`LogFormatter` class and making
`LOG_FORMATTER` point to your new class.
#### *class* scrapy.logformatter.LogFormatter
Class for generating log messages for different actions.
All methods must return a dictionary listing the parameters `level`, `msg`
and `args` which are going to be used for constructing the log message when
calling `logging.log`.
Dictionary keys for the method outputs:
* `level` is the log level for that action, you can use those from the
[python logging library](https://docs.python.org/3/library/logging.html) :
`logging.DEBUG`, `logging.INFO`, `logging.WARNING`, `logging.ERROR`
and `logging.CRITICAL`.
* `msg` should be a string that can contain different formatting placeholders.
This string, formatted with the provided `args`, is going to be the long message
for that action.
* `args` should be a tuple or dict with the formatting placeholders for `msg`.
The final log message is computed as `msg % args`.
Users can define their own `LogFormatter` class if they want to customize how
each action is logged or if they want to omit it entirely. In order to omit
logging an action the method must return `None`.
Here is an example on how to create a custom log formatter to lower the severity level of
the log message when an item is dropped from the pipeline:
```python
class PoliteLogFormatter(logformatter.LogFormatter):
def dropped(self, item, exception, response, spider):
return {
"level": logging.INFO, # lowering the level from logging.WARNING
"msg": "Dropped: %(exception)s" + os.linesep + "%(item)s",
"args": {
"exception": exception,
"item": item,
},
}
```
##### crawled(request: Request, response: Response, spider: Spider) → LogFormatterResult
Logs a message when the crawler finds a webpage.
##### scraped(item: Any, response: Response | Failure | None, spider: Spider) → LogFormatterResult
Logs a message when an item is scraped by a spider.
##### dropped(item: Any, exception: BaseException, response: Response | Failure | None, spider: Spider) → LogFormatterResult
Logs a message when an item is dropped while it is passing through the item pipeline.
##### item_error(item: Any, exception: BaseException, response: Response | Failure | None, spider: Spider) → LogFormatterResult
Logs a message when an item causes an error while it is passing
through the item pipeline.
##### spider_error(failure: Failure, request: Request, response: Response | Failure, spider: Spider) → LogFormatterResult
Logs an error message from a spider.
##### download_error(failure: Failure, request: Request, spider: Spider, errmsg: str | None = None) → LogFormatterResult
Logs a download error message from a spider (typically coming from
the engine).
#### Advanced customization
Because Scrapy uses stdlib logging module, you can customize logging using
all features of stdlib logging.
For example, let’s say you’re scraping a website which returns many
HTTP 404 and 500 responses, and you want to hide all messages like this:
```
2016-12-16 22:00:06 [scrapy.spidermiddlewares.httperror] INFO: Ignoring
response <500 https://quotes.toscrape.com/page/1-34/>: HTTP status code
is not handled or not allowed
```
The first thing to note is a logger name - it is in brackets:
`[scrapy.spidermiddlewares.httperror]`. If you get just `[scrapy]` then
`LOG_SHORT_NAMES` is likely set to True; set it to False and re-run
the crawl.
Next, we can see that the message has INFO level. To hide it
we should set logging level for `scrapy.spidermiddlewares.httperror`
higher than INFO; next level after INFO is WARNING. It could be done
e.g. in the spider’s `__init__` method:
```python
import logging
import scrapy
class MySpider(scrapy.Spider):
# ...
def __init__(self, *args, **kwargs):
logger = logging.getLogger("scrapy.spidermiddlewares.httperror")
logger.setLevel(logging.WARNING)
super().__init__(*args, **kwargs)
```
If you run this spider again then INFO messages from
`scrapy.spidermiddlewares.httperror` logger will be gone.
You can also filter log records by `LogRecord` data. For
example, you can filter log records by message content using a substring or
a regular expression. Create a `logging.Filter` subclass
and equip it with a regular expression pattern to
filter out unwanted messages:
```python
import logging
import re
class ContentFilter(logging.Filter):
def filter(self, record):
match = re.search(r"\d{3} [Ee]rror, retrying", record.message)
if match:
return False
```
A project-level filter may be attached to the root
handler created by Scrapy, this is a wieldy way to
filter all loggers in different parts of the project
(middlewares, spider, etc.):
```python
import logging
import scrapy
class MySpider(scrapy.Spider):
# ...
def __init__(self, *args, **kwargs):
for handler in logging.root.handlers:
handler.addFilter(ContentFilter())
```
Alternatively, you may choose a specific logger
and hide it without affecting other loggers:
```python
import logging
import scrapy
class MySpider(scrapy.Spider):
# ...
def __init__(self, *args, **kwargs):
logger = logging.getLogger("my_logger")
logger.addFilter(ContentFilter())
```
### scrapy.utils.log module
#### scrapy.utils.log.configure_logging(settings: Settings | dict[str, Any] | None = None, install_root_handler: bool | None = None) → None
Initialize logging defaults for Scrapy.
* **Parameters:**
* **settings** (dict, `Settings` object or `None`) – settings used to create and configure a handler for the
root logger (default: None).
* **install_root_handler** (*bool*) – whether to install root logging handler
(default: the `LOG_INSTALL_ROOT_HANDLER` setting)
This function does:
- Route warnings and twisted logging through Python standard logging
- Assign DEBUG and ERROR level to Scrapy and Twisted loggers respectively
- Route stdout to log if LOG_STDOUT setting is True
When installing a root logging handler, this function also creates a
handler for the root logger according to given settings (see
topics-logging-settings). You can override default options using
`settings` argument. When `settings` is empty or None, defaults are
used.
`configure_logging` is automatically called when using Scrapy commands
or `CrawlerProcess`, but needs to be called explicitly
when running custom scripts using `CrawlerRunner`.
In that case, its usage is not required but it’s recommended.
Another option when running custom scripts is to manually configure the logging.
To do this you can use `logging.basicConfig()` to set a basic root handler.
Note that `CrawlerProcess` automatically calls `configure_logging`,
so it is recommended to only use `logging.basicConfig()` together with
`CrawlerRunner`.
This is an example on how to redirect `INFO` or higher messages to a file:
```python
import logging
logging.basicConfig(
filename="log.txt", format="%(levelname)s: %(message)s", level=logging.INFO
)
```
Refer to run-from-script for more details about using Scrapy this
way.
Source: /topics/stats.md
## Stats Collection
Scrapy provides a convenient facility for collecting stats in the form of
key/values, where values are often counters. The facility is called the Stats
Collector, and can be accessed through the `stats`
attribute of the topics-api-crawler, as illustrated by the examples in
the topics-stats-usecases section below.
The Stats Collector API is always available, so you can always use it (to
increment or set new stat keys), regardless
of whether the stats collection is enabled or not. If it’s disabled, the API
will still work but it won’t collect anything. This is aimed at simplifying the
stats collector usage: you should spend no more than one line of code for
collecting stats in your spider, Scrapy extension, or whatever code you’re
using the Stats Collector from.
Another feature of the Stats Collector is that it’s very efficient (when
enabled) and extremely efficient (almost unnoticeable) when disabled.
See topics-stats-reference below for the stats that Scrapy sets.
### Common Stats Collector uses
Access the stats collector through the `stats`
attribute. Here is an example of an extension that access stats:
```python
class ExtensionThatAccessStats:
def __init__(self, stats):
self.stats = stats
@classmethod
def from_crawler(cls, crawler):
return cls(crawler.stats)
```
Set stat value:
```python
stats.set_value("hostname", socket.gethostname())
```
Increment stat value:
```python
stats.inc_value("custom_count")
```
Set stat value only if greater than previous:
```python
stats.max_value("max_items_scraped", value)
```
Set stat value only if lower than previous:
```python
stats.min_value("min_free_memory_percent", value)
```
Get stat value:
```pycon
>>> stats.get_value("custom_count")
1
```
Get all stats:
```pycon
>>> stats.get_stats()
{'custom_count': 1, 'start_time': datetime.datetime(2009, 7, 14, 21, 47, 28, 977139)}
```
### Available Stats Collectors
Besides the basic `StatsCollector` there are other Stats Collectors
available in Scrapy which extend the basic Stats Collector. You can select
which Stats Collector to use through the `STATS_CLASS` setting. The
default Stats Collector used is the `MemoryStatsCollector`.
#### StatsCollector
#### *class* scrapy.statscollectors.StatsCollector(crawler: Crawler)
The base stats collector that other stats collectors are based on.
##### get_value(key: str, default: Any = None, spider: Spider | None = None) → Any
Return the value of the *key* stat, or *default* if it is not set.
##### get_stats(spider: Spider | None = None) → StatsT
Return all stats as a dict.
##### set_value(key: str, value: Any, spider: Spider | None = None) → None
Set the *key* stat to *value*.
##### set_stats(stats: StatsT, spider: Spider | None = None) → None
Replace all stats with *stats*.
##### inc_value(key: str, count: int = 1, start: int = 0, spider: Spider | None = None) → None
Increment the *key* stat by *count*, or set it to *start* if it is
not set.
##### max_value(key: str, value: Any, spider: Spider | None = None) → None
Set the *key* stat to *value* if it is not set or lower than
*value*.
##### min_value(key: str, value: Any, spider: Spider | None = None) → None
Set the *key* stat to *value* if it is not set or higher than
*value*.
##### clear_stats(spider: Spider | None = None) → None
Clear all stats.
##### open_spider(spider: Spider | None = None) → None
Called when the spider is opened.
##### close_spider(spider: Spider | None = None, reason: str | None = None) → None
Called when the spider is closed.
#### MemoryStatsCollector
#### *class* scrapy.statscollectors.MemoryStatsCollector(crawler: Crawler)
A simple stats collector that keeps the stats of the last scraping run
(for each spider) in memory, after they’re closed. The stats can be
accessed through the `spider_stats` attribute, which is a dict keyed
by spider name.
This is the default stats collector used in Scrapy.
##### spider_stats *: dict[str, StatsT]*
A dict of dicts (keyed by spider name) containing the stats of the last
scraping run for each spider.
#### DummyStatsCollector
#### *class* scrapy.statscollectors.DummyStatsCollector(crawler: Crawler)
A stats collector which does nothing but is very efficient (because it
does nothing). This stats collector can be set via the
`STATS_CLASS` setting, to disable stats collection in order to
improve performance. However, the performance penalty of stats collection
is usually marginal compared to other Scrapy workload like parsing pages.
### Built-in stats reference
Scrapy sets the following stats. Components other than
those built into Scrapy may set additional stats; see their documentation.
Stat keys that contain a `{placeholder}` below stand for a family of stats,
one per actual value of the placeholder.
> [!NOTE]
> Most stats are set by a specific component, and are only present if that component is enabled and
> its code path is reached. A stat that is missing from
> `get_stats()` output is
> equivalent to a counter of 0.
`depth/request_ignored_count`
: Number of requests dropped for exceeding `DEPTH_LIMIT`.
<br/>
Set by `DepthMiddleware`.
`downloader/exception_count`
: Number of exceptions raised while downloading requests.
<br/>
Set by `DownloaderStats`.
`downloader/exception_type_count/{exception_type}`
: Number of exceptions raised while downloading requests, per exception type,
where `{exception_type}` is the import path of the exception class, e.g.
`twisted.internet.error.DNSLookupError`.
<br/>
Set by `DownloaderStats`.
`downloader/request_bytes`
: Total size, in bytes, of the requests sent, counting the request line, the
headers and the body. As with `downloader/request_count`, requests
served from the cache are also counted.
<br/>
It is an approximation, reconstructed from each `Request`
object instead of measured on the wire, so it does not account for the
actual bytes that the download handler sends, e.g. transport-level overhead.
<br/>
Set by `DownloaderStats`.
`downloader/request_count`
: Number of requests sent.
<br/>
Requests that `HttpCacheMiddleware`
serves from the cache are also counted, even though they are never sent,
because it handles requests after
`DownloaderStats`.
<br/>
Set by `DownloaderStats`.
`downloader/request_method_count/{method}`
: Number of requests sent, per HTTP method, e.g. `GET` or `POST`. As with
`downloader/request_count`, requests served from the cache are also
counted.
<br/>
Set by `DownloaderStats`.
`downloader/response_bytes`
: Total size, in bytes, of the responses received, counting the status line,
the headers and the body. It covers the same responses as
`downloader/response_count`.
<br/>
The body is counted as received, i.e. still compressed for responses that
used `Content-Encoding`, because
`DownloaderStats` handles
responses before
`HttpCompressionMiddleware`
decompresses them. See `httpcompression/response_bytes` for
decompressed sizes.
<br/>
Set by `DownloaderStats`.
`downloader/response_count`
: Number of responses received.
<br/>
It counts responses that `HttpCacheMiddleware`
serves from the cache, even though they do not come from the network, and
responses that a downloader middleware consumes before they reach your
spider, e.g. redirect responses that `RedirectMiddleware`
turns into new requests. Compare with `response_received_count`.
<br/>
Set by `DownloaderStats`.
`downloader/response_status_count/{status_code}`
: Number of responses received, per HTTP status code, e.g. `200` or
`404`. It covers the same responses as `downloader/response_count`.
<br/>
Set by `DownloaderStats`.
`dupefilter/filtered`
: Number of requests dropped as duplicates.
<br/>
Set by `RFPDupeFilter`.
`elapsed_time_seconds`
: Time, as a `float`, in seconds, between the `spider_opened`
and the `spider_closed` signals.
<br/>
Set by `CoreStats`.
`feedexport/failed_count/{storage}`
: Number of feeds that could not be stored, per
storage backend, where `{storage}`
is the class name of the storage backend, e.g. `FileFeedStorage`.
`feedexport/success_count/{storage}`
: Number of feeds stored successfully, per
storage backend, where `{storage}`
is the class name of the storage backend, e.g. `FileFeedStorage`.
`file_count`
: Number of files handled by the media pipelines.
`file_status_count/{status}`
: Number of files handled by the media pipelines, per status, where `{status}` is one of:
<br/>
- `downloaded`: the file was downloaded.
- `cached`: the file came from the
`HttpCacheMiddleware`
cache.
- `uptodate`: the file was already in the storage backend and had not
expired, so it was not downloaded again.
`finish_reason`
: String indicating why the crawl finished. It matches the *reason* argument
of the `spider_closed` signal.
<br/>
Scrapy uses the following reasons:
<br/>
- `cancelled`: the spider was closed without a more specific reason,
e.g. because `CloseSpider` was raised without
one.
- `closespider_errorcount`: see `CLOSESPIDER_ERRORCOUNT`.
- `closespider_itemcount`: see `CLOSESPIDER_ITEMCOUNT`.
- `closespider_pagecount`: see `CLOSESPIDER_PAGECOUNT`.
- `closespider_pagecount_no_item`: see
`CLOSESPIDER_PAGECOUNT_NO_ITEM`.
- `closespider_timeout`: see `CLOSESPIDER_TIMEOUT`.
- `closespider_timeout_no_item`: see
`CLOSESPIDER_TIMEOUT_NO_ITEM`.
- `finished`: the spider became idle with no pending requests, i.e. it
finished normally.
- `memusage_exceeded`: see `MEMUSAGE_LIMIT_MB`.
- `shutdown`: the crawl was interrupted, e.g. by a system signal such
as `SIGINT` (`Ctrl`-`C`).
- `start_error`: `start()` raised an exception, so
some start requests may never have been sent,
see start-error.
<br/>
Third-party components and your own code may use any other reason, e.g. by
raising `CloseSpider` with it.
<br/>
Set by `CoreStats`.
`finish_time`
: Timezone-aware `datetime` object, in UTC, indicating when
the `spider_closed` signal was sent.
<br/>
Set by `CoreStats`.
`httpcache/errorrecovery`
: Number of times that a stale cached response was used because downloading a
fresh response raised an exception.
<br/>
Set by `HttpCacheMiddleware`.
`httpcache/firsthand`
: Number of responses that were downloaded without a matching cache entry to
validate against, i.e. responses for requests counted in
`httpcache/miss`.
<br/>
It is lower than `httpcache/miss` when some of those requests yield
no response, either because they are dropped (see
`httpcache/ignore`) or because their download fails.
<br/>
Set by `HttpCacheMiddleware`.
`httpcache/hit`
: Number of requests served from the cache.
<br/>
Set by `HttpCacheMiddleware`.
`httpcache/ignore`
: Number of requests dropped because they were not in the cache and
`HTTPCACHE_IGNORE_MISSING` is `True`.
<br/>
Set by `HttpCacheMiddleware`.
`httpcache/invalidate`
: Number of times that a cached response failed validation and was replaced
with a freshly downloaded response.
<br/>
Set by `HttpCacheMiddleware`.
`httpcache/miss`
: Number of requests for which no cache entry could be read, either because
there was none or because reading it failed, in which case the request is
also counted in `httpcache/retrieve_error`. Those requests are
downloaded (see `httpcache/firsthand`), or dropped if
`HTTPCACHE_IGNORE_MISSING` is `True` (see
`httpcache/ignore`).
<br/>
Requests with a stale cache entry are not counted here; see
`httpcache/revalidate` and `httpcache/invalidate`.
<br/>
Set by `HttpCacheMiddleware`.
`httpcache/retrieve_error`
: Number of cache entries that could not be read, and hence were treated as
cache misses. Those requests are also counted in `httpcache/miss`.
<br/>
Set by `HttpCacheMiddleware`.
`httpcache/revalidate`
: Number of times that a cached response was successfully validated against
the target server, and hence used instead of the fresh response.
<br/>
Set by `HttpCacheMiddleware`.
`httpcache/store`
: Number of responses stored in the cache.
<br/>
Set by `HttpCacheMiddleware`.
`httpcache/uncacheable`
: Number of responses not stored in the cache because the
`HTTPCACHE_POLICY` did not allow it.
<br/>
Every response considered for caching is counted either here or in
`httpcache/store`, so `httpcache/store + httpcache/uncacheable`
equals `httpcache/firsthand + httpcache/invalidate`.
<br/>
Set by `HttpCacheMiddleware`.
`httpcompression/response_bytes`
: Total size, in bytes, of decompressed response bodies, counting only the
body and only responses that were actually decompressed. Compare with
`downloader/response_bytes`.
<br/>
Set by
`HttpCompressionMiddleware`.
`httpcompression/response_count`
: Number of decompressed responses.
<br/>
Set by
`HttpCompressionMiddleware`.
`httperror/response_ignored_count`
: Number of responses dropped because of their HTTP status code.
<br/>
Set by `HttpErrorMiddleware`.
`httperror/response_ignored_status_count/{status_code}`
: Number of responses dropped because of their HTTP status code, per HTTP
status code, e.g. `404`.
<br/>
Set by `HttpErrorMiddleware`.
`item_dropped_count`
: Number of items dropped by an item pipeline, i.e. number of times that the
`item_dropped` signal was sent.
<br/>
Set by `CoreStats`.
`item_dropped_reasons_count/{exception}`
: Number of items dropped, per exception, where `{exception}` is the class
name of the exception that caused the item to be dropped.
<br/>
Only `DropItem` and its subclasses drop items, and
each one is counted under its own class name, e.g.
`item_dropped_reasons_count/DropItem` for
`DropItem` itself and
`item_dropped_reasons_count/MyDropItem` for a `MyDropItem` subclass of
it. Any other exception raised by an item pipeline triggers the `item_error` signal instead of
`item_dropped`, and is not counted here or in
`item_dropped_count`.
<br/>
Set by `CoreStats`.
`item_scraped_count`
: Number of items that passed all item pipelines, i.e. number of times that the
`item_scraped` signal was sent.
<br/>
Set by `CoreStats`.
`items_per_minute`
: Average number of items scraped per minute during the crawl.
<br/>
It is `None` if the crawl took less than a minute.
<br/>
Set by `LogStats`.
`log_count/{level}`
: Number of log messages, per logging level name, e.g. `INFO` or
`WARNING`.
<br/>
Only messages that the `LOG_LEVEL` setting allows are counted.
<br/>
Set by `LogCount`.
`memdebug/gc_garbage_count`
: Number of objects in `gc.garbage` when the spider is closed.
<br/>
Set by `MemoryDebugger`, which requires
`MEMDEBUG_ENABLED` to be `True`.
`memdebug/live_refs/{cls}`
: Number of live objects of class `{cls}` when the spider is closed, as
reported by trackref, e.g.
`memdebug/live_refs/HtmlResponse`.
<br/>
Only set for classes with at least 1 live object.
<br/>
Set by `MemoryDebugger`, which requires
`MEMDEBUG_ENABLED` to be `True`.
`memusage/limit_reached`
: `1` if memory usage exceeded `MEMUSAGE_LIMIT_MB`, which also
stops the crawl.
<br/>
Set by `MemoryUsage`.
`memusage/max`
: Maximum peak memory usage, in bytes, observed during the crawl.
<br/>
Set by `MemoryUsage`.
`memusage/startup`
: Peak memory usage, in bytes, when the engine started.
<br/>
Set by `MemoryUsage`.
`memusage/warning_reached`
: `1` if memory usage exceeded `MEMUSAGE_WARNING_MB`.
<br/>
Set by `MemoryUsage`.
`offsite/domains`
: Number of distinct domains for which at least 1 request was dropped for
being offsite.
<br/>
Set by `OffsiteMiddleware`.
`offsite/filtered`
: Number of requests dropped for being offsite.
<br/>
Set by `OffsiteMiddleware`.
`request_depth_count/{depth}`
: Number of requests scheduled at depth `{depth}`, e.g.
`request_depth_count/2`.
<br/>
Set by `DepthMiddleware`, which
requires `DEPTH_STATS_VERBOSE` to be `True` for this stat.
`request_depth_max`
: Maximum depth reached.
<br/>
Set by `DepthMiddleware`.
`response_received_count`
: Number of responses received, i.e. number of times that the
`response_received` signal was sent.
<br/>
Unlike `downloader/response_count`, it does not count responses that
a downloader middleware consumes before they reach the engine, e.g.
redirect responses that `RedirectMiddleware`
turns into new requests. Both count responses that `HttpCacheMiddleware`
serves from the cache.
<br/>
Set by `CoreStats`.
`responses_per_minute`
: Average number of responses received per minute during the crawl.
<br/>
It is `None` if the crawl took less than a minute.
<br/>
Set by `LogStats`.
`retry/count`
: Number of requests retried.
<br/>
Set by `get_retry_request()`, which
`RetryMiddleware` uses.
`retry/max_reached`
: Number of requests that were not retried because they had already been
retried `RETRY_TIMES` times.
<br/>
Set by `get_retry_request()`, which
`RetryMiddleware` uses.
`retry/reason_count/{reason}`
: Number of requests retried, per reason, e.g.
`retry/reason_count/twisted.internet.error.TimeoutError` or
`retry/reason_count/504 Gateway Time-out`.
<br/>
Set by `get_retry_request()`, which
`RetryMiddleware` uses.
> [!NOTE]
> Code calling
> `get_retry_request()` may pass a
> custom *stats_base_key*, in which case `retry` is replaced with that key
> in the 3 stats above.
`robotstxt/exception_count/{exception_type}`
: Number of exceptions raised while downloading `robots.txt` files, per
exception type, where `{exception_type}` is the string representation of
the exception class, e.g. `<class
'twisted.internet.error.DNSLookupError'>`.
<br/>
Set by
`RobotsTxtMiddleware`.
`robotstxt/forbidden`
: Number of requests dropped for being disallowed by `robots.txt`.
<br/>
Set by
`RobotsTxtMiddleware`.
`robotstxt/request_count`
: Number of `robots.txt` files requested, i.e. 1 per network location for
which at least 1 request was sent.
<br/>
Set by
`RobotsTxtMiddleware`.
`robotstxt/response_count`
: Number of `robots.txt` responses received.
<br/>
Set by
`RobotsTxtMiddleware`.
`robotstxt/response_status_count/{status_code}`
: Number of `robots.txt` responses received, per HTTP status code, e.g.
`404`.
<br/>
Set by
`RobotsTxtMiddleware`.
`scheduler/dequeued`
: Number of requests read from the scheduler.
`scheduler/dequeued/disk`
: Number of requests read from the disk queue of the scheduler.
`scheduler/dequeued/memory`
: Number of requests read from the memory queue of the scheduler.
`scheduler/enqueued`
: Number of requests stored into the scheduler.
`scheduler/enqueued/disk`
: Number of requests stored into the disk queue of the scheduler.
`scheduler/enqueued/memory`
: Number of requests stored into the memory queue of the scheduler.
`scheduler/unserializable`
: Number of requests that could not be stored into the disk queue of the
scheduler because they could not be
serialized, and hence were stored into the
memory queue instead.
`spider_exceptions/count`
: Number of unhandled exceptions raised by spider callbacks or by
`start()`.
<br/>
Set by the engine and the scraper.
`spider_exceptions/{exception}`
: Same as `spider_exceptions/count`, per exception, where
`{exception}` is the class name of the exception, e.g.
`spider_exceptions/ValueError`.
<br/>
Set by the engine and the scraper.
`start_time`
: Timezone-aware `datetime` object, in UTC, indicating when
the `spider_opened` signal was sent.
<br/>
Set by `CoreStats`.
`urllength/request_ignored_count`
: Number of requests dropped for having a URL longer than
`URLLENGTH_LIMIT`.
<br/>
Set by `UrlLengthMiddleware`.
Source: /topics/telnetconsole.md
## Telnet Console
Scrapy comes with a built-in telnet console for inspecting and controlling a
Scrapy running process. The telnet console is just a regular python shell
running inside the Scrapy process, so you can do literally anything from it.
The telnet console is a built-in Scrapy extension which comes enabled by default, but you can also
disable it if you want. For more information about the extension itself see
topics-extensions-ref-telnetconsole.
> [!WARNING]
> It is not secure to use telnet console via public networks, as telnet
> doesn’t provide any transport-layer security. Having username/password
> authentication doesn’t change that.
>
> Intended usage is connecting to a running Scrapy spider locally
> (spider process and telnet client are on the same machine)
> or over a secure connection (VPN, SSH tunnel).
> Please avoid using telnet console over insecure connections,
> or disable it completely using `TELNETCONSOLE_ENABLED` option.
> [!NOTE]
> This feature is not supported when `TWISTED_REACTOR_ENABLED` is `False`.
> [!NOTE]
> ### See also
>
> security-telnet, using-mcp-server
### How to access the telnet console
The telnet console listens on the first available TCP port from the range
defined in the `TELNETCONSOLE_PORT` setting, which defaults to
`[6023, 6073]`. To access the console you need to type:
```none
telnet localhost 6023
Trying localhost...
Connected to localhost.
Escape character is '^]'.
Username:
Password:
>>>
```
By default, the username is `scrapy` and the password is autogenerated. The
autogenerated password can be seen on Scrapy logs like the example below:
```none
2018-10-16 14:35:21 [scrapy.extensions.telnet] INFO: Telnet Password: 16f92501e8a59326
```
The default username and password can be overridden by the settings
`TELNETCONSOLE_USERNAME` and `TELNETCONSOLE_PASSWORD`.
> [!WARNING]
> Username and password provide only a limited protection, as telnet
> is not using secure transport - by default traffic is not encrypted
> even if username and password are set.
You need the telnet program which comes installed by default in Windows, and
most Linux distros.
### Available variables in the telnet console
The telnet console is like a regular Python shell running inside the Scrapy
process, so you can do anything from it including importing new modules, etc.
However, the telnet console comes with some default variables defined for
convenience:
| Shortcut | Description |
|--------------|---------------------------------------------------------|
| `crawler` | the Scrapy Crawler (`scrapy.crawler.Crawler` object) |
| `engine` | Crawler.engine attribute |
| `spider` | the active spider |
| `extensions` | the Extension Manager (Crawler.extensions attribute) |
| `stats` | the Stats Collector (Crawler.stats attribute) |
| `settings` | the Scrapy settings object (Crawler.settings attribute) |
| `est` | print a report of the engine status |
| `prefs` | for memory debugging (see topics-leaks) |
| `p` | a shortcut to the `pprint.pprint()` function |
### Telnet console usage examples
Here are some example tasks you can do with the telnet console:
#### View engine status
You can use the `est()` method provided by the console to quickly show the
engine status:
```none
telnet localhost 6023
>>> est()
Execution engine status
time()-engine.start_time : 8.62972998619
len(engine.downloader.active) : 16
engine.scraper.is_idle() : False
engine.spider.name : followall
engine.spider_is_idle() : False
engine._slot.closing : False
len(engine._slot.inprogress) : 16
len(engine.scheduler.dqs or []) : 0
len(engine.scheduler.mqs) : 92
len(engine.scraper.slot.queue) : 0
len(engine.scraper.slot.active) : 0
engine.scraper.slot.active_size : 0
engine.scraper.slot.itemproc_size : 0
engine.scraper.slot.needs_backout() : False
```
#### Pause, resume and stop the Scrapy engine
To pause:
```none
telnet localhost 6023
>>> engine.pause()
>>>
```
To resume:
```none
telnet localhost 6023
>>> engine.unpause()
>>>
```
To stop:
```none
telnet localhost 6023
>>> engine.stop()
Connection closed by foreign host.
```
### Telnet Console signals
#### scrapy.extensions.telnet.update_telnet_vars(telnet_vars)
Sent just before the telnet console is opened. You can hook up to this
signal to add, remove or update the variables that will be available in the
telnet local namespace. In order to do that, you need to update the
`telnet_vars` dict in your handler.
* **Parameters:**
**telnet_vars** (*dict*) – the dict of telnet variables
### Telnet settings
These are the settings that control the telnet console’s behaviour:
#### TELNETCONSOLE_PORT
Default: `[6023, 6073]`
The port range to use for the telnet console. If set to `None`, a dynamically
assigned port is used.
#### TELNETCONSOLE_HOST
Default: `'127.0.0.1'`
The interface the telnet console should listen on
> [!NOTE]
> ### See also
>
> security-telnet
#### TELNETCONSOLE_USERNAME
Default: `'scrapy'`
The username used for the telnet console
#### TELNETCONSOLE_PASSWORD
Default: `None`
The password used for the telnet console, default behaviour is to have it
autogenerated
Source: /faq.md
## Frequently Asked Questions
### How does Scrapy compare to BeautifulSoup or lxml?
[BeautifulSoup](https://www.crummy.com/software/BeautifulSoup/) and [lxml](https://lxml.de/) are libraries for parsing HTML and XML. Scrapy is
an application framework for writing web spiders that crawl web sites and
extract data from them.
Scrapy provides a built-in mechanism for extracting data (called
selectors) but you can easily use [BeautifulSoup](https://www.crummy.com/software/BeautifulSoup/)
(or [lxml](https://lxml.de/)) instead, if you feel more comfortable working with them. After
all, they’re just parsing libraries which can be imported and used from any
Python code.
In other words, comparing [BeautifulSoup](https://www.crummy.com/software/BeautifulSoup/) (or [lxml](https://lxml.de/)) to Scrapy is like
comparing [jinja2](https://palletsprojects.com/projects/jinja/) to [Django](https://www.djangoproject.com/).
### Can I use Scrapy with BeautifulSoup?
Yes, you can.
As mentioned above, [BeautifulSoup](https://www.crummy.com/software/BeautifulSoup/) can be used
for parsing HTML responses in Scrapy callbacks.
You just have to feed the response’s body into a `BeautifulSoup` object
and extract whatever data you need from it.
Here’s an example spider using BeautifulSoup API, with `lxml` as the HTML parser:
```python
from bs4 import BeautifulSoup
import scrapy
class ExampleSpider(scrapy.Spider):
name = "example"
allowed_domains = ["example.com"]
start_urls = ("http://www.example.com/",)
def parse(self, response):
# use lxml to get decent HTML parsing speed
soup = BeautifulSoup(response.text, "lxml")
yield {"url": response.url, "title": soup.h1.string}
```
> [!NOTE]
> `BeautifulSoup` supports several HTML/XML parsers.
> See [BeautifulSoup’s official documentation](https://www.crummy.com/software/BeautifulSoup/bs4/doc/#specifying-the-parser-to-use) on which ones are available.
### Did Scrapy “steal” X from Django?
Probably, but we don’t like that word. We think [Django](https://www.djangoproject.com/) is a great open source
project and an example to follow, so we’ve used it as an inspiration for
Scrapy.
We believe that, if something is already done well, there’s no need to reinvent
it. This concept, besides being one of the foundations for open source and free
software, not only applies to software but also to documentation, procedures,
policies, etc. So, instead of going through each problem ourselves, we choose
to copy ideas from those projects that have already solved them properly, and
focus on the real problems we need to solve.
We’d be proud if Scrapy serves as an inspiration for other projects. Feel free
to steal from us!
### Does Scrapy work with HTTP proxies?
Yes. Support for HTTP proxies is provided through the HTTP Proxy downloader
middleware. See
`HttpProxyMiddleware`.
### Does Scrapy work with SOCKS proxies?
Yes, when using
`HttpxDownloadHandler`. See
`HttpProxyMiddleware` and the
handler documentation.
### How can I scrape an item with attributes in different pages?
See callback-data.
### How can I simulate a user login in my spider?
See topics-request-response-ref-request-userlogin.
### Does Scrapy crawl in breadth-first or depth-first order?
DFO by default, but other orders are possible.
### My Scrapy crawler has memory leaks. What can I do?
See topics-leaks.
Also, Python has a builtin memory leak issue which is described in
topics-leaks-without-leaks.
### How can I make Scrapy consume less memory?
See previous question.
### How can I prevent memory errors due to many allowed domains?
If you have a spider with a long list of `allowed_domains`
(e.g. 50,000+), consider replacing the default
`OffsiteMiddleware` downloader
middleware with a custom downloader middleware that requires less memory. For example:
- If your domain names are similar enough, use your own regular expression
instead of joining the strings in `allowed_domains` into
a complex regular expression.
- If you can meet the installation requirements, use [pyre2](https://github.com/andreasvc/pyre2) instead of
Python’s [re](https://docs.python.org/3/library/re.html) to compile your URL-filtering regular expression. See
[#1908](https://github.com/scrapy/scrapy/issues/1908).
See also [other suggestions at StackOverflow](https://stackoverflow.com/q/36440681).
> [!NOTE]
> Remember to disable
> `scrapy.downloadermiddlewares.offsite.OffsiteMiddleware` when you
> enable your custom implementation:
>
> ```python
> DOWNLOADER_MIDDLEWARES = {
> "scrapy.downloadermiddlewares.offsite.OffsiteMiddleware": None,
> "myproject.middlewares.CustomOffsiteMiddleware": 50,
> }
> ```
### Can I use Basic HTTP Authentication in my spiders?
Yes, see `HttpAuthMiddleware`.
### Why does Scrapy download pages in English instead of my native language?
Try changing the default [Accept-Language](https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.4) request header by overriding the
`DEFAULT_REQUEST_HEADERS` setting.
### Where can I find some example Scrapy projects?
See intro-examples.
### Can I run a spider without creating a project?
Yes. You can use the `runspider` command. For example, if you have a
spider written in a `my_spider.py` file you can run it with:
```
scrapy runspider my_spider.py
```
See `runspider` command for more info.
### I get “Filtered offsite request” messages. How can I fix them?
Those messages (logged with `DEBUG` level) don’t necessarily mean there is a
problem, so you may not need to fix them.
Those messages are thrown by
`OffsiteMiddleware`, which is a
downloader middleware (enabled by default) whose purpose is to filter out
requests to domains outside the ones covered by the spider.
### What is the recommended way to deploy a Scrapy crawler in production?
See topics-deploy.
### Can I use JSON for large exports?
It’ll depend on how large your output is. See this warning in `JsonItemExporter`
documentation.
### Can I return (Twisted) deferreds from signal handlers?
Some signals support returning deferreds from their handlers, others don’t. See
the topics-signals-ref to know which ones.
### What does the response status code 999 mean?
999 is a custom response status code used by some sites to throttle requests.
Try slowing down the crawling speed by using a download delay of `2` (or
higher) for the affected domains, with the `DOWNLOAD_SLOTS` setting:
```python
DOWNLOAD_SLOTS = {
"example.com": {"delay": 2},
}
```
Or by setting a global download delay in your project with the
`DOWNLOAD_DELAY` setting.
### Can I call `pdb.set_trace()` from my spiders to debug them?
Yes, but you can also use the Scrapy shell which allows you to quickly analyze
(and even modify) the response being processed by your spider, which is, quite
often, more useful than plain old `pdb.set_trace()`.
For more info see topics-shell-inspect-response.
You can also connect to live crawls, which is the
best way to debug running spiders without adding breakpoints to their code in
advance.
### Simplest way to dump all my scraped items into a JSON/CSV/XML file?
To dump into a JSON file:
```
scrapy crawl myspider -O items.json
```
To dump into a CSV file:
```
scrapy crawl myspider -O items.csv
```
To dump into an XML file:
```
scrapy crawl myspider -O items.xml
```
For more information see topics-feed-exports
### What’s this huge cryptic `__VIEWSTATE` parameter used in some forms?
The `__VIEWSTATE` parameter is used in sites built with ASP.NET/VB.NET. For
more info on how it works see [this page](https://metacpan.org/release/ECARROLL/HTML-TreeBuilderX-ASP_NET-0.09/view/lib/HTML/TreeBuilderX/ASP_NET.pm). Also, here’s an [example spider](https://github.com/AmbientLighter/rpn-fas/blob/master/fas/spiders/rnp.py)
which scrapes one of these sites.
### What’s the best way to parse big XML/CSV data feeds?
Parsing big feeds with XPath selectors can be problematic since they need to
build the DOM of the entire feed in memory, and this can be quite slow and
consume a lot of memory.
In order to avoid parsing all the entire feed at once in memory, you can use
the `xmliter_lxml()` and
`csviter()` functions. In fact, this is what
`XMLFeedSpider` and
`CSVFeedSpider` use.
#### scrapy.utils.iterators.xmliter_lxml(obj: Response | str | bytes, nodename: str, namespace: str | None = None, prefix: str = 'x') → Iterator[Selector]
#### scrapy.utils.iterators.csviter(obj: Response | str | bytes, delimiter: str | None = None, headers: list[str] | None = None, encoding: str | None = None, quotechar: str | None = None) → Iterator[dict[str, str]]
Returns an iterator of dictionaries from the given csv object
obj can be:
- a Response object
- a unicode string
- a string encoded as utf-8
delimiter is the character used to separate fields on the given obj.
headers is an iterable that when provided offers the keys
for the returned dictionaries, if not the first row is used.
quotechar is the character used to enclosure fields on the given obj.
### Does Scrapy manage cookies automatically?
Yes, Scrapy receives and keeps track of cookies sent by servers, and sends them
back on subsequent requests, like any regular web browser does.
For more info see cookies.
### How can I see the cookies being sent and received from Scrapy?
Enable the `COOKIES_DEBUG` setting.
### How can I instruct a spider to stop itself?
Raise the `CloseSpider` exception from a callback. For
more info see: `CloseSpider`.
### How can I prevent my Scrapy bot from getting banned?
See bans.
### Should I use spider arguments or settings to configure my spider?
Both spider arguments and settings
can be used to configure your spider. There is no strict rule that mandates to
use one or the other, but settings are more suited for parameters that, once
set, don’t change much, while spider arguments are meant to change more often,
even on each spider run and sometimes are required for the spider to run at all
(for example, to set the start url of a spider).
To illustrate with an example, assuming you have a spider that needs to log
into a site to scrape data, and you only want to scrape data from a certain
section of the site (which varies each time). In that case, the credentials to
log in would be settings, while the url of the section to scrape would be a
spider argument.
### I’m scraping an XML document and my XPath selector doesn’t return any items
You may need to remove namespaces. See removing-namespaces.
### How to split an item into multiple items in an item pipeline?
Item pipelines cannot yield multiple items per
input item. Create a spider middleware
instead, and use its
`process_spider_output()`
method for this purpose. For example:
```python
from copy import deepcopy
from itemadapter import ItemAdapter
from scrapy import Request
class MultiplyItemsMiddleware:
def process_spider_output(self, response, result):
for item_or_request in result:
if isinstance(item_or_request, Request):
yield item_or_request
continue
adapter = ItemAdapter(item_or_request)
for _ in range(adapter["multiply_by"]):
yield deepcopy(item_or_request)
```
### Does Scrapy support IPv6 addresses?
Yes, but when using
`HTTP11DownloadHandler` or
`H2DownloadHandler` you need to
set `TWISTED_DNS_RESOLVER` to `scrapy.resolver.CachingHostnameResolver`.
Note that by doing so, you lose the ability to set a specific timeout for DNS requests
(the value of the `DNS_TIMEOUT` setting is ignored).
### How to deal with `<class 'ValueError'>: filedescriptor out of range in select()` exceptions?
This issue [has been reported](https://github.com/scrapy/scrapy/issues/2905) to appear when running broad crawls in macOS, where the default
Twisted reactor was `twisted.internet.selectreactor.SelectReactor` at that time.
If you have switched to this reactor using the `TWISTED_REACTOR` setting you can switch
to a different one in the same way.
### How can I cancel the download of a given response?
In some situations, it might be useful to stop the download of a certain response.
For instance, sometimes you can determine whether or not you need the full contents
of a response by inspecting its headers or the first bytes of its body. In that case,
you could save resources by attaching a handler to the `bytes_received`
or `headers_received` signals and raising a
`StopDownload` exception. Please refer to the
topics-stop-response-download topic for additional information and examples.
### How can I make a blank request?
```python
from scrapy import Request
blank_request = Request("data:,")
```
In this case, the URL is set to a data URI scheme. Data URLs allow you to include data
inline within web pages, similar to external resources. The “data:” scheme with an empty
content (“,”) essentially creates a request to a data URL without any specific content.
### Running `runspider` I get `error: No spider found in file: <filename>`
This may happen if your Scrapy project has a spider module with a name that
conflicts with the name of one of the [Python standard library modules](https://docs.python.org/3/py-modindex.html), such
as `csv.py` or `os.py`, or any [Python package](https://pypi.org/) that you have installed.
See [#2680](https://github.com/scrapy/scrapy/issues/2680).
Source: /topics/agents.md
## Using Scrapy with coding agents
If you use coding agents to work with Scrapy projects, we recommend installing
the official [agent plugin](https://github.com/scrapy/scrapy-agent-plugin). To install it, point your agent to the plugin
repository URL (read the plugin documentation for more information):
```
https://github.com/scrapy/scrapy-agent-plugin
```
This plugin adds a skill that helps agents build and troubleshoot Scrapy
projects, and configures the official [Scrapy MCP server](https://github.com/scrapy/scrapy-mcp-official), which allows agents
to connect to Scrapy crawls.
### Using the MCP server
> [!WARNING]
> Added in version 2.19.0.
The [Scrapy MCP server](https://github.com/scrapy/scrapy-mcp-official) can connect to running local crawls that have the
`RemoteControl` extension enabled. Here are some tasks that are
possible thanks to it:
- Checking on the progress of a crawl, by looking at the current values of
stats and, when that’s not enough, by inspecting core
objects such as the scheduler directly.
- Investigating why a crawl is slow, doesn’t make requests, or makes requests
without producing items, by looking at its runtime state, settings, and
source code. It’s also easy to see how the runtime state (e.g. the stats or
the queue state) changes over a set amount of time, by reading it twice in
the same request with an `asyncio.sleep()` call in between.
See optimize-bottleneck for more ideas about investigating slow
crawls.
- Studying memory usage and debugging memory leaks, as described in
topics-leaks.
- Modifying the runtime state, configuration, and even code of a crawl. This is
fragile and not always possible, but it may save a broken crawl that is
expensive or impossible to restart without losing data and that cannot be
fixed in any other way.
Source: /topics/debug.md
## Debugging Spiders
This document explains the most common techniques for debugging spiders.
Consider the following Scrapy spider below:
```python
import scrapy
from myproject.items import MyItem
class MySpider(scrapy.Spider):
name = "myspider"
start_urls = (
"http://example.com/page1",
"http://example.com/page2",
)
def parse(self, response):
# <processing code not shown>
# collect `item_urls`
for item_url in item_urls:
yield scrapy.Request(item_url, self.parse_item)
def parse_item(self, response):
# <processing code not shown>
item = MyItem()
# populate `item` fields
# and extract item_details_url
yield scrapy.Request(
item_details_url, self.parse_details, cb_kwargs={"item": item}
)
def parse_details(self, response, item):
# populate more `item` fields
return item
```
Basically this is a simple spider which parses two pages of items (the
start_urls). Items also have a details page with additional information, so we
use the `cb_kwargs` functionality of `Request` to pass a
partially populated item.
### Parse Command
The most basic way of checking the output of your spider is to use the
`parse` command. It allows to check the behaviour of different parts
of the spider at the method level. It has the advantage of being flexible and
simple to use, but does not allow debugging code inside a method.
In order to see the item scraped from a specific url:
```none
$ scrapy parse --spider=myspider -c parse_item -d 2 <item_url>
[ ... scrapy log lines crawling example.com spider ... ]
>>> STATUS DEPTH LEVEL 2 <<<
# Scraped Items ------------------------------------------------------------
[{'url': <item_url>}]
# Requests -----------------------------------------------------------------
[]
```
Using the `--verbose` or `-v` option we can see the status at each depth level:
```none
$ scrapy parse --spider=myspider -c parse_item -d 2 -v <item_url>
[ ... scrapy log lines crawling example.com spider ... ]
>>> DEPTH LEVEL: 1 <<<
# Scraped Items ------------------------------------------------------------
[]
# Requests -----------------------------------------------------------------
[<GET item_details_url>]
>>> DEPTH LEVEL: 2 <<<
# Scraped Items ------------------------------------------------------------
[{'url': <item_url>}]
# Requests -----------------------------------------------------------------
[]
```
Checking items scraped from a single start_url, can also be easily achieved
using:
```none
$ scrapy parse --spider=myspider -d 3 'http://example.com/page1'
```
### Scrapy Shell
While the `parse` command is very useful for checking behaviour of a
spider, it is of little help to check what happens inside a callback, besides
showing the response received and the output. How to debug the situation when
`parse_details` sometimes receives no item?
Fortunately, the `shell` is your bread and butter in this case (see
topics-shell-inspect-response):
```python
from scrapy.shell import inspect_response
def parse_details(self, response, item=None):
if item:
# populate more `item` fields
return item
else:
inspect_response(response, self)
```
See also: topics-shell-inspect-response.
### Scrapy MCP server
You can use the Scrapy MCP server to connect a coding
agent to a running crawl, so that it can check the progress of the crawl and
investigate or even modify its runtime state.
### Open in browser
Sometimes you just want to see how a certain response looks in a browser, you
can use the `open_in_browser()` function for that:
#### scrapy.utils.response.open_in_browser(response: Response, \_openfunc: Callable[[str], Any] = <function open>) → Any
Open *response* in a local web browser, adjusting the [base tag](https://www.w3schools.com/tags/tag_base.asp) for
external links to work, e.g. so that images and styles are displayed.
For example:
```python
from scrapy.utils.response import open_in_browser
def parse_details(self, response):
if "item name" not in response.text:
open_in_browser(response)
```
On the Windows Subsystem for Linux, set the `BROWSER` environment
variable to [wslview](https://github.com/wslutilities/wslu) to open the
response in a Windows browser, which cannot read Linux paths otherwise.
### Logging
Logging is another useful option for getting information about your spider run.
Although not as convenient, it comes with the advantage that the logs will be
available in all future runs should they be necessary again:
```python
def parse_details(self, response, item=None):
if item:
# populate more `item` fields
return item
else:
self.logger.warning("No item received for %s", response.url)
```
For more information, check the topics-logging section.
### Inspecting live traffic
Sometimes it’s important to see what exactly was sent to the server or received
from it, such as header values, formatting and order (Scrapy cannot log this,
as underlying HTTP libraries produce the final values for request headers and
canonicalize response ones) or TLS handshake details. There are two ways to see
and log the real traffic of a running spider:
- Capture the traffic with a tool such as [Wireshark](https://www.wireshark.org/). As your requests likely
use TLS, you will need to decrypt the traffic (see the [Wireshark TLS
documentation](https://wiki.wireshark.org/TLS) for detailed instructions). You will need the encryption
key which you can save as described in debug-tls. As this way of
capturing traffic is passive, it cannot interfere with the spider.
- Use [mitmproxy](https://mitmproxy.org/) between the spider and the server, as described below. This
is easier to set up and in addition to inspecting the traffic allows
modifying it, but it’s not passive: there is now a connection between
Scrapy and mitmproxy and another one between mitmproxy and the server
instead of a direct connection between Scrapy and the server. Due to this,
low-level connection behavior is different from normal crawls, which may
change the server behavior, and you cannot easily use mitmproxy and regular
proxies in the same crawl.
#### Using mitmdump
You should refer to the mitmproxy documentation for more details, additional
interception modes and advanced features but here is one simple way to use it.
First, run a `mitmdump` instance (it will use the port 8080 by default),
asking it to log the traffic details on the terminal (`--flow-detail 2` will
log headers but not bodies):
```shell
mitmdump --flow-detail 2
```
Then configure `http://127.0.0.1:8080` as a proxy in your spider using
`HttpProxyMiddleware`:
```shell
https_proxy=http://127.0.0.1:8080 scrapy crawl myspider
```
To inspect only some requests, set their `proxy` meta key instead.
### Decrypting TLS traffic
Scrapy writes the session keys of its HTTPS connections to the file that the
`SSLKEYLOGFILE` environment variable points to, using the [NSS key log
format](https://firefox-source-docs.mozilla.org/security/nss/legacy/key_log_format/index.html) that traffic analysis tools such as Wireshark understand.
> [!WARNING]
> Added in version 2.18.0.
```shell
SSLKEYLOGFILE=/tmp/sslkeylog scrapy crawl myspider
```
> [!WARNING]
> Anyone who can read the key log file can decrypt the traffic of
> the connections recorded in it, including any credentials that they carry.
### Visual Studio Code
To debug spiders with Visual Studio Code you can use the following `launch.json`:
```json
{
"version": "0.1.0",
"configurations": [
{
"name": "Python: Launch Scrapy Spider",
"type": "python",
"request": "launch",
"module": "scrapy",
"args": [
"runspider",
"${file}"
],
"console": "integratedTerminal"
}
]
}
```
Also, make sure you enable “User Uncaught Exceptions”, to catch exceptions in
your Scrapy spider.
Source: /topics/contracts.md
## Spiders Contracts
Testing spiders can get particularly annoying and while nothing prevents you
from writing unit tests the task gets cumbersome quickly. Scrapy offers an
integrated way of testing your spiders by the means of contracts.
> [!WARNING]
> Changed in version 2.19.0: Added support for callbacks defined with `async def`, including
> asynchronous generators.
This allows you to test each callback of your spider by hardcoding a sample url
and check various constraints for how the callback processes the response. Each
contract is prefixed with an `@` and included in the docstring. See the
following example:
```python
def parse(self, response):
"""
This function parses a sample response. Some contracts are mingled
with this docstring.
@url http://www.example.com/s?field-keywords=selfish+gene
@returns items 1 16
@returns requests 0 0
@scrapes Title Author Year Price
"""
```
You can use the following contracts:
#### *class* scrapy.contracts.default.UrlContract(method: Callable[..., Any], \*args: Any)
Sets (`@url`) the sample URL used when checking the other contract
conditions of a callback.
This contract is mandatory: callbacks lacking it are ignored when running
the checks.
```none
@url url
```
#### *class* scrapy.contracts.default.CallbackKeywordArgumentsContract(method: Callable[..., Any], \*args: Any)
Sets (`@cb_kwargs`) the `cb_kwargs`
attribute of the sample request.
Its value must be a valid JSON dictionary.
```none
@cb_kwargs {"arg1": "value1", "arg2": "value2", ...}
```
#### *class* scrapy.contracts.default.MetadataContract(method: Callable[..., Any], \*args: Any)
Sets (`@meta`) the `meta` attribute of the
sample request.
Its value must be a valid JSON dictionary.
```none
@meta {"arg1": "value1", "arg2": "value2", ...}
```
#### *class* scrapy.contracts.default.MethodContract(method: Callable[..., Any], \*args: Any)
Sets (`@method`) the `method` of the
sample request.
> [!WARNING]
> Added in version 2.19.0.
```none
@method POST
```
#### *class* scrapy.contracts.default.BodyContract(method: Callable[..., Any], \*args: Any)
Sets (`@body`) the `body` of the sample
request.
> [!WARNING]
> Added in version 2.19.0.
```none
@body field1=value1&field2=value2
```
#### *class* scrapy.contracts.default.HeaderContract(method: Callable[..., Any], \*args: Any)
Sets (`@header`) a header of the sample request. Use one line per
header.
> [!WARNING]
> Added in version 2.19.0.
```none
@header name value
```
#### *class* scrapy.contracts.default.CookieContract(method: Callable[..., Any], \*args: Any)
Sets (`@cookie`) a cookie of the sample request. Use one line per
cookie.
> [!WARNING]
> Added in version 2.19.0.
```none
@cookie name value
```
#### *class* scrapy.contracts.default.ReturnsContract(\*args: Any, \*\*kwargs: Any)
Sets (`@returns`) lower and upper bounds for the items and requests
returned by a callback.
The upper bound is optional:
```none
@returns item(s)|request(s) [min [max]]
```
For example:
```none
@returns request
@returns request 2
@returns request 2 10
@returns request 0 10
```
Set both bounds to the same value to require an exact number:
```none
@returns request 2 2
```
#### *class* scrapy.contracts.default.ScrapesContract(method: Callable[..., Any], \*args: Any)
Checks (`@scrapes`) that all items returned by a callback have the
specified fields.
```none
@scrapes field_1 field_2 ...
```
Use the `check` command to run the contract checks. It ignores
`ITEM_PIPELINES` and `FEEDS`, since contracts check the
output of callbacks instead of sending it to item processing; use the `-s`
command-line option to set them back for a check run.
> [!WARNING]
> Changed in version 2.18.0: `ITEM_PIPELINES` and `FEEDS` are now ignored.
### Custom Contracts
If you find you need more power than the built-in Scrapy contracts you can
create and load your own contracts in the project by using the
`SPIDER_CONTRACTS` setting:
```python
SPIDER_CONTRACTS = {
"myproject.contracts.ResponseCheck": 10,
"myproject.contracts.ItemValidate": 10,
}
```
Each contract must inherit from `Contract` and can
override three methods:
#### *class* scrapy.contracts.Contract(method: Callable[..., Any], \*args: Any)
Base class for custom contracts.
*method* is the callback function to which the contract is associated.
*args* is the list of arguments passed into the docstring, separated by
whitespace.
Subclasses may override `adjust_request_args()`, and define a
`pre_process` method or a `post_process` method, or both.
##### adjust_request_args(args: dict[str, Any]) → dict[str, Any]
Receive a `dict` with the default arguments for the sample request
and return it, either unmodified or with changes.
`Request` is used by default, but this can be changed
with the `request_cls` attribute. If multiple contracts in the chain
define this attribute, the last one is used.
##### pre_process(response)
This allows hooking in various checks on the response received from the
sample request, before it’s being passed to the callback.
##### post_process(output)
This allows processing the output of the callback. Iterators are
converted to lists before being passed to this hook.
Raise `ContractFail` from
`pre_process` or
`post_process` if expectations are not met:
#### *class* scrapy.exceptions.ContractFail
Error raised in case of a failing contract
Here is a demo contract which checks the presence of a custom header in the
response received:
```python
from scrapy.contracts import Contract
from scrapy.exceptions import ContractFail
class HasHeaderContract(Contract):
"""
Demo contract which checks the presence of a custom header
@has_header X-CustomHeader
"""
name = "has_header"
def pre_process(self, response):
for header in self.args:
if header not in response.headers:
raise ContractFail("X-CustomHeader not present")
```
### Detecting check runs
When `scrapy check` is running, the `SCRAPY_CHECK` environment variable is
set to the `true` string. You can use `os.environ` to perform any change to
your spiders or your settings when `scrapy check` is used:
```python
import os
import scrapy
class ExampleSpider(scrapy.Spider):
name = "example"
def __init__(self):
if os.environ.get("SCRAPY_CHECK"):
pass # Do some scraper adjustments when a check is running
```
Source: /topics/practices.md
## Common Practices
This section documents common practices when using Scrapy. These are things
that cover many topics and don’t often fall into any other specific section.
### Run Scrapy from a script
You can use the API to run Scrapy from a script, instead of
the typical way of running Scrapy via `scrapy crawl`.
Remember that Scrapy requires a Twisted reactor or (with
`TWISTED_REACTOR_ENABLED` set to `False`) an asyncio event loop, so
you need to run one of those in your script for it to work (helpers described
below can do it for you).
The first utility you can use to run your spiders is
`scrapy.crawler.AsyncCrawlerProcess` or
`scrapy.crawler.CrawlerProcess`. These classes will start a Twisted
reactor for you, configuring the logging and setting shutdown handlers. These
classes are the ones used by all Scrapy commands. They have similar
functionality, differing in their asynchronous API style:
`AsyncCrawlerProcess` returns coroutines from its
asynchronous methods while `CrawlerProcess` returns
`Deferred` objects.
Here’s an example showing how to run a single spider with it.
```python
import scrapy
from scrapy.crawler import AsyncCrawlerProcess
class MySpider(scrapy.Spider):
# Your spider definition
...
process = AsyncCrawlerProcess(
settings={
"FEEDS": {
"items.json": {"format": "json"},
},
}
)
process.crawl(MySpider)
process.start() # the script will block here until the crawling is finished
```
You can define settings within the dictionary passed
to `AsyncCrawlerProcess`. Make sure to check the
`AsyncCrawlerProcess`
documentation to get acquainted with its usage details.
If you are inside a Scrapy project there are some additional helpers you can
use to import those components within the project. You can automatically import
your spiders passing their name to
`AsyncCrawlerProcess`, and use
`scrapy.utils.project.get_project_settings()` to get a
`Settings` instance with your project settings.
What follows is a working example of how to do that, using the [testspiders](https://github.com/scrapinghub/testspiders)
project as example.
```python
from scrapy.crawler import AsyncCrawlerProcess
from scrapy.utils.project import get_project_settings
process = AsyncCrawlerProcess(get_project_settings())
# 'followall' is the name of one of the spiders of the project.
process.crawl("followall", domain="scrapy.org")
process.start() # the script will block here until the crawling is finished
```
There’s another Scrapy utility that provides more control over the crawling
process: `scrapy.crawler.AsyncCrawlerRunner` or
`scrapy.crawler.CrawlerRunner`. These classes are thin wrappers
that encapsulate some simple helpers to run multiple crawlers, but they won’t
start or interfere with existing reactors in any way. Just like
`scrapy.crawler.AsyncCrawlerProcess` and
`scrapy.crawler.CrawlerProcess` they differ in their asynchronous API
style.
When using these classes the reactor should be explicitly run after scheduling
your spiders. It’s recommended that you use
`AsyncCrawlerRunner` or
`CrawlerRunner` instead of
`AsyncCrawlerProcess` or
`CrawlerProcess` if your application is already using
Twisted and you want to run Scrapy in the same reactor.
If you want to stop the reactor or run any other code right after the spider
finishes you can do that after the task returned from
`AsyncCrawlerRunner.crawl()`
completes (or the Deferred returned from `CrawlerRunner.crawl()` fires). In the simplest case you can also
use `twisted.internet.task.react()` to start and stop the reactor, though
it may be easier to just use `AsyncCrawlerProcess` or
`CrawlerProcess` instead.
Here’s an example of using `AsyncCrawlerRunner` together
with simple reactor management code:
```python
import scrapy
from scrapy.crawler import AsyncCrawlerRunner
from scrapy.utils.defer import deferred_f_from_coro_f
from scrapy.utils.log import configure_logging
from scrapy.utils.reactor import install_reactor
from twisted.internet.task import react
class MySpider(scrapy.Spider):
# Your spider definition
...
async def crawl(_):
configure_logging({"LOG_FORMAT": "%(levelname)s: %(message)s"})
runner = AsyncCrawlerRunner()
await runner.crawl(MySpider) # completes when the spider finishes
install_reactor("twisted.internet.asyncioreactor.AsyncioSelectorReactor")
react(deferred_f_from_coro_f(crawl))
```
Same example but using `CrawlerRunner` and a
different reactor (`AsyncCrawlerRunner` only works
with `AsyncioSelectorReactor`):
```python
import scrapy
from scrapy.crawler import CrawlerRunner
from scrapy.utils.log import configure_logging
from scrapy.utils.reactor import install_reactor
from twisted.internet.task import react
class MySpider(scrapy.Spider):
custom_settings = {
"TWISTED_REACTOR": "twisted.internet.epollreactor.EPollReactor",
}
# Your spider definition
...
def crawl(_):
configure_logging({"LOG_FORMAT": "%(levelname)s: %(message)s"})
runner = CrawlerRunner()
d = runner.crawl(MySpider)
return d # this Deferred fires when the spider finishes
install_reactor("twisted.internet.epollreactor.EPollReactor")
react(crawl)
```
> [!NOTE]
> ### See also
>
> twisted:core/howto/reactor-basics
And here are examples of using these classes with
`TWISTED_REACTOR_ENABLED` set to `False`.
Simple usage of `AsyncCrawlerProcess`:
```python
import scrapy
from scrapy.crawler import AsyncCrawlerProcess
class MySpider(scrapy.Spider):
# Your spider definition
...
process = AsyncCrawlerProcess(
settings={
"TWISTED_REACTOR_ENABLED": False,
}
)
process.crawl(MySpider)
process.start() # the script will block here until the crawling is finished
```
With `TWISTED_REACTOR_ENABLED=False` you can use several instances of
`AsyncCrawlerProcess` in the same process:
```python
import scrapy
from scrapy.crawler import AsyncCrawlerProcess
class MySpider(scrapy.Spider):
# Your spider definition
...
process1 = AsyncCrawlerProcess(
settings={
"TWISTED_REACTOR_ENABLED": False,
}
)
process1.crawl(MySpider)
process1.start()
process2 = AsyncCrawlerProcess(
settings={
"TWISTED_REACTOR_ENABLED": False,
}
)
process2.crawl(MySpider)
process2.start()
```
Using `asyncio.run()` with `AsyncCrawlerRunner`:
```python
import asyncio
import scrapy
from scrapy.crawler import AsyncCrawlerRunner
from scrapy.utils.log import configure_logging
class MySpider(scrapy.Spider):
# Your spider definition
...
async def main():
configure_logging({"LOG_FORMAT": "%(levelname)s: %(message)s"})
runner = AsyncCrawlerRunner(settings={"TWISTED_REACTOR_ENABLED": False})
await runner.crawl(MySpider) # completes when the spider finishes
asyncio.run(main())
```
### Running spiders inside existing applications
You may want to run Scrapy spiders inside an existing application. In simple
cases (e.g. task queues that spawn a process for every task, or applications
that can execute tasks synchronously in the same process) you can use the same
approach as for standalone scripts (see run-from-script). More complex
cases, e.g. asynchronous web applications, have additional caveats and
limitations.
If the application runs its own Twisted reactor, you can use
`AsyncCrawlerRunner` or
`CrawlerRunner` to run spiders using this reactor, see
run-from-script for examples.
If the application doesn’t run a Twisted reactor or an asyncio event loop (for
example, a Django web app deployed with a WSGI server such as uWSGI), you can
use `AsyncCrawlerProcess` with
`TWISTED_REACTOR_ENABLED` set to `False`, so that Scrapy starts and
stops an asyncio event loop for every spider run:
```python
import scrapy
from django.http import HttpResponse
from scrapy.crawler import AsyncCrawlerProcess
class MySpider(scrapy.Spider):
# Your spider definition
...
def crawl_view(request):
process = AsyncCrawlerProcess(settings={"TWISTED_REACTOR_ENABLED": False})
process.crawl(MySpider)
process.start() # returns when the spider finishes
return HttpResponse("Crawling finished")
```
If the application runs its own asyncio event loop (for example, a Django web
app deployed with an ASGI server such as uvicorn), you can use
`AsyncCrawlerRunner` with
`TWISTED_REACTOR_ENABLED` set to `False`, so that Scrapy uses the
existing event loop:
```python
import scrapy
from django.http import HttpResponse
from scrapy.crawler import AsyncCrawlerRunner
class MySpider(scrapy.Spider):
# Your spider definition
...
async def crawl_view(request):
runner = AsyncCrawlerRunner(settings={"TWISTED_REACTOR_ENABLED": False})
await runner.crawl(MySpider) # completes when the spider finishes
return HttpResponse("Crawling finished")
```
> [!NOTE]
> Running Scrapy without a Twisted reactor is experimental and has
> some limitations, described in asyncio-without-reactor.
### Running spiders in Jupyter notebooks
You can run Scrapy spiders in Jupyter notebooks. You need to use
`AsyncCrawlerRunner` with
`TWISTED_REACTOR_ENABLED` set to `False` for this, so that Scrapy
uses the event loop provided by the notebook kernel. As
`AsyncCrawlerRunner` doesn’t configure logging, and you
most likely want to see the spider log in the notebook, you should call
`scrapy.utils.log.configure_logging()`. Here is a full example, which
supports rerunning both as a single cell and as separate cells:
```python
from scrapy import Spider
from scrapy.crawler import AsyncCrawlerRunner
from scrapy.utils.log import configure_logging
configure_logging()
class BooksSpider(Spider):
name = "books"
start_urls = ["https://books.toscrape.com"]
def parse(self, response):
for book in response.css("h3"):
yield {"title": book.css("a::attr(title)").get()}
runner = AsyncCrawlerRunner({"TWISTED_REACTOR_ENABLED": False})
await runner.crawl(BooksSpider)
```
> [!NOTE]
> Running Scrapy without a Twisted reactor is experimental and has
> some limitations, described in asyncio-without-reactor.
### Running multiple spiders in the same process
By default, Scrapy runs a single spider per process when you run `scrapy
crawl`. However, Scrapy supports running multiple spiders per process using
the internal API.
Each call to `crawl()` creates its own `Crawler`,
with its own instances of the downloader and spider middlewares and its own
resolved settings, including spider settings. Nothing from one of these is shared with the other
spiders running in the same process.
Here is an example that runs multiple spiders simultaneously:
```python
import scrapy
from scrapy.crawler import AsyncCrawlerProcess
from scrapy.utils.project import get_project_settings
class MySpider1(scrapy.Spider):
# Your first spider definition
...
class MySpider2(scrapy.Spider):
# Your second spider definition
...
settings = get_project_settings()
process = AsyncCrawlerProcess(settings)
process.crawl(MySpider1)
process.crawl(MySpider2)
process.start() # the script will block here until all crawling jobs are finished
```
Same example using `AsyncCrawlerRunner`:
```python
import scrapy
from scrapy.crawler import AsyncCrawlerRunner
from scrapy.utils.defer import deferred_f_from_coro_f
from scrapy.utils.log import configure_logging
from scrapy.utils.reactor import install_reactor
from twisted.internet.task import react
class MySpider1(scrapy.Spider):
# Your first spider definition
...
class MySpider2(scrapy.Spider):
# Your second spider definition
...
async def crawl(_):
configure_logging({"LOG_FORMAT": "%(levelname)s: %(message)s"})
runner = AsyncCrawlerRunner()
runner.crawl(MySpider1)
runner.crawl(MySpider2)
await runner.join() # completes when both spiders finish
install_reactor("twisted.internet.asyncioreactor.AsyncioSelectorReactor")
react(deferred_f_from_coro_f(crawl))
```
Same example but running the spiders sequentially by awaiting until each one
finishes before starting the next one:
```python
import scrapy
from scrapy.crawler import AsyncCrawlerRunner
from scrapy.utils.defer import deferred_f_from_coro_f
from scrapy.utils.log import configure_logging
from scrapy.utils.reactor import install_reactor
from twisted.internet.task import react
class MySpider1(scrapy.Spider):
# Your first spider definition
...
class MySpider2(scrapy.Spider):
# Your second spider definition
...
async def crawl(_):
configure_logging({"LOG_FORMAT": "%(levelname)s: %(message)s"})
runner = AsyncCrawlerRunner()
await runner.crawl(MySpider1)
await runner.crawl(MySpider2)
install_reactor("twisted.internet.asyncioreactor.AsyncioSelectorReactor")
react(deferred_f_from_coro_f(crawl))
```
> [!NOTE]
> When running multiple spiders in the same process, logging
> settings and reactor settings
> should not have a different value per spider, and pre-crawler
> settings cannot be defined per spider.
Every other setting applies to each crawler separately. This includes
concurrency and politeness settings, such as `CONCURRENT_REQUESTS`,
`CONCURRENT_REQUESTS_PER_DOMAIN` and `DOWNLOAD_DELAY`, and
AutoThrottle also throttles each crawler
separately. When crawling simultaneously, divide those values by the number of
crawlers to keep the combined load on your hardware and on target websites
unchanged.
Because of this, running the same spider several times in the same process
multiplies those limits instead of increasing crawling capacity. To crawl
faster, raise `CONCURRENT_REQUESTS` on a single crawler.
> [!NOTE]
> ### See also
>
> run-from-script.
### Distributed crawls
Scrapy doesn’t provide any built-in facility for running crawls in a distributed
(multi-server) manner. However, there are some ways to distribute crawls, which
vary depending on how you plan to distribute them.
If you have many spiders, the obvious way to distribute the load is to setup
many Scrapyd instances and distribute spider runs among those.
If you instead want to run a single (big) spider through many machines, what
you usually do is partition the URLs to crawl and send them to each separate
spider. Here is a concrete example:
First, you prepare the list of URLs to crawl and put them into separate
files/urls:
```
http://somedomain.com/urls-to-crawl/spider1/part1.list
http://somedomain.com/urls-to-crawl/spider1/part2.list
http://somedomain.com/urls-to-crawl/spider1/part3.list
```
Then you fire a spider run on 3 different Scrapyd servers. The spider would
receive a (spider) argument `part` with the number of the partition to
crawl:
```
curl http://scrapy1.mycompany.com:6800/schedule.json -d project=myproject -d spider=spider1 -d part=1
curl http://scrapy2.mycompany.com:6800/schedule.json -d project=myproject -d spider=spider1 -d part=2
curl http://scrapy3.mycompany.com:6800/schedule.json -d project=myproject -d spider=spider1 -d part=3
```
### Reducing startup time in large projects
When running a spider with `scrapy crawl`, Scrapy loads all modules listed in
`SPIDER_MODULES` to find the target spider. In large projects with
many spiders, this can noticeably increase startup time and memory usage.
To avoid loading every spider module, override `SPIDER_MODULES` on the
command line to point only to the module that contains the spider you want to
run:
```shell
scrapy crawl myspider -s SPIDER_MODULES=myproject.spiders.myspider
```
Because `SPIDER_MODULES` is a list setting, you can include multiple
modules by separating them with commas.
### Avoiding getting banned
Websites tell regular visitors and crawlers apart by how their traffic looks:
the headers it carries, how fast it arrives, how many requests come from the
same place. Traffic that stands out can be blocked even when the crawling
itself would be welcome.
Where the website allows crawling, the most effective thing you can do is make
yourself known: set `USER_AGENT` to a value that identifies you and
lets its owners reach you, so that they can ask you to adjust your crawler
rather than block it.
Where that is not enough, the following make your traffic resemble that of a
regular visitor:
* rotate your user agent among those of common browsers, so that your requests
do not all look alike (search the web for an up-to-date list)
* disable cookies (see `COOKIES_ENABLED`), so that a session
identifier does not tie all your requests together
* space out your requests, 2 seconds apart or more, with the
`DOWNLOAD_DELAY` setting, to keep your pace closer to that of a
person browsing
* where possible, read pages from [Common Crawl](https://commoncrawl.org/), which sends no traffic to
the website at all
* spread your requests over a pool of IP addresses, so that none of them
accounts for your whole crawl. For example, the free [Tor project](https://www.torproject.org/) or paid
services like [ProxyMesh](https://proxymesh.com/).
* match the TLS behavior of a browser: some websites respond differently
depending on the TLS version of the client, which you can adjust with the
`DOWNLOAD_TLS_MIN_VERSION` and `DOWNLOAD_TLS_MAX_VERSION`
settings.
* let a service take care of all of the above, such as [Zyte API](https://docs.zyte.com/zyte-api/get-started.md), which
provides a [Scrapy plugin](https://github.com/scrapy-plugins/scrapy-zyte-api) and additional
features, like [AI web scraping](https://www.zyte.com/ai-web-scraping/)
If your crawler still gets blocked, consider contacting [commercial support](https://www.scrapy.org/companies).
### Static analysis
Consider using scrapy-lint, a linter for Scrapy
projects that detects common mistakes and anti-patterns.
### Connecting to live crawls
It’s useful to be able to connect to live long-running crawls, either to check
their progress in detail or to investigate problems with them. Scrapy provides
the following tools for this:
- Telnet console: connect to a crawl process with
a telnet client and execute Python code inside it.
- Scrapy MCP server: point a coding agent to a crawl
process so that it can execute Python code inside it.
Source: /topics/security.md
## Security
Scrapy defaults are optimized for web scraping, not for the security posture
that you might expect from software that handles untrusted input or runs in a
shared or exposed environment. Some common security practices are unnecessary
for many scraping use cases, and a few can even prevent valid ones (for
example, sites that you must scrape may use misconfigured TLS certificates or
serve content over unencrypted protocols).
This page highlights the Scrapy defaults that have security implications, so
that you can make an informed decision about whether to keep them, and explains
how to harden them along with the trade-offs involved.
> [!NOTE]
> None of the options below are silver bullets. Which of them make sense
> depends on your threat model: whether the URLs you crawl come from trusted
> sources, whether the machine running Scrapy is exposed to a network you do
> not control, whether the data you handle is sensitive, and so on.
### Treat responses as untrusted input
Regardless of any setting, remember that response data comes from servers you
do not control, even when you trust the site you are crawling, as responses may
be tampered with in transit or the server itself may be compromised.
Never pass response data to functions that can execute code or otherwise act on
their input in an unsafe way, such as `eval()`, `exec()`, or
`pickle.loads()`, and be careful when writing response data to paths
derived from the response itself.
### Memory use when parsing responses
Parsing a response with selectors builds an in-memory
tree of the whole response body, which takes several times as much memory as
the body itself. Scrapy parses without the size limits that libxml2 applies by
default, so the size of that tree is bound only by the size of the response, as
controlled by `DOWNLOAD_MAXSIZE` (default: 1 GiB).
XML entities are left unresolved, so the tree stays proportional to the
response body even for input crafted as an [XML bomb](https://lxml.de/FAQ.html#is-lxml-vulnerable-to-xml-bombs). A server can still
make a crawler allocate a lot of memory by returning a very large response,
though, so if you know the size of the responses you care about, lower the
limit:
```python
DOWNLOAD_MAXSIZE = 32 * 1024 * 1024 # 32 MiB
```
* **Pro:** a server cannot make the crawler allocate more memory than the limit
allows, whether by returning a large response or by crafting one that is
expensive to parse.
* **Con:** you can no longer scrape sites that legitimately serve responses
above the limit, as those responses are dropped.
#### Parser limits
The limits that libxml2 applies by default, such as 256 nesting levels and
10 MB per text node, can be restored by overriding
`selector` in a response subclass and swapping
responses in a downloader middleware:
```python
from functools import cached_property
from scrapy import Selector
from scrapy.http import HtmlResponse
class LimitedHtmlResponse(HtmlResponse):
@cached_property
def selector(self):
return Selector(self, huge_tree=False)
class LimitedParsingMiddleware:
def process_response(self, request, response, spider):
if isinstance(response, HtmlResponse):
return response.replace(cls=LimitedHtmlResponse)
return response
```
Do the same with `XmlResponse` if you also parse XML.
These limits apply per node, so `DOWNLOAD_MAXSIZE` remains your bound
on total memory: a response made of many small elements is parsed in full and
uses as much memory either way.
* **Pro:** deeply nested responses, and responses with very large individual
nodes, become cheaper to parse.
* **Con:** parsing stops at those limits without raising, so a legitimate page
that exceeds them yields incomplete data and no error.
### TLS connections
#### Certificate verification
By default Scrapy does **not** verify the TLS certificate of HTTPS servers, as
controlled by the `DOWNLOAD_VERIFY_CERTIFICATES` setting (default:
`False`).
This default favors reach over security: many sites that are otherwise fine to
scrape have expired, self-signed, or otherwise invalid certificates, and
verifying certificates would make requests to them fail.
If the integrity of the connection matters to you (for example, to detect
man-in-the-middle attacks), set:
```python
DOWNLOAD_VERIFY_CERTIFICATES = True
```
* **Pro:** requests to servers with invalid or untrusted certificates fail
instead of silently succeeding, protecting you from some man-in-the-middle
attacks.
* **Con:** you can no longer scrape sites with misconfigured certificates
without re-disabling verification for them.
#### Protocol versions and ciphers
You can restrict the TLS protocol versions that Scrapy accepts through the
`DOWNLOAD_TLS_MIN_VERSION` and `DOWNLOAD_TLS_MAX_VERSION`
settings, e.g. to reject obsolete protocol versions.
By default Scrapy uses the OpenSSL `DEFAULT` cipher list
(`DOWNLOADER_CLIENT_TLS_CIPHERS`), which favors compatibility and still
allows some older, weaker ciphers. Set it to `None` to instead use the curated
cipher list of the underlying TLS implementation (Twisted), which excludes weak
ciphers:
```python
DOWNLOADER_CLIENT_TLS_CIPHERS = None
```
* **Pro:** connections that would negotiate a weak cipher fail instead of
succeeding.
* **Con:** you can no longer connect to servers that only support the excluded
ciphers.
### Unencrypted protocols
By default Scrapy enables download handlers for unencrypted protocols, namely
`http://` and `ftp://` (see `DOWNLOAD_HANDLERS_BASE`). Data sent
and received over these protocols, including any credentials, travels in plain
text and can be read or modified by anyone on the network path.
If you only crawl over encrypted protocols, you can disable the unencrypted
ones so that no request can accidentally be sent unencrypted:
```python
DOWNLOAD_HANDLERS = {
"http": None,
"ftp": None,
}
```
* **Pro:** a misconfigured or maliciously-redirected request cannot leak data
over an unencrypted connection, as such requests fail instead.
* **Con:** you can no longer crawl resources that are only available over those
protocols.
Note that disabling the `http` handler also prevents plain-HTTP requests that
result from following an `http://` redirect or link, which is often the point
of disabling it.
### Local and non-network resources
By default Scrapy enables download handlers for the `file://` and `data:`
schemes (see `DOWNLOAD_HANDLERS_BASE`). The `file://` handler reads
arbitrary files from the local filesystem, limited only by the permissions of
the process running Scrapy.
This is convenient (for example, to parse a local HTML file), but it is a risk
if any of the URLs you schedule come from an untrusted source: a crafted
`file:///etc/passwd` URL could read local files.
If you do not need them, disable these handlers:
```python
DOWNLOAD_HANDLERS = {
"file": None,
"data": None,
}
```
* **Pro:** crawled URLs cannot be used to read local files or inline data.
* **Con:** you can no longer fetch `file://` or `data:` URLs.
More generally, if you crawl URLs from untrusted sources, consider validating
their schemes (and, where applicable, their hosts) before scheduling requests,
to avoid server-side request forgery (SSRF) and similar issues.
### Remote control server
Scrapy enables the remote control HTTP server
(`scrapy.extensions.remote_control.RemoteControl`) by default
(`REMOTE_CONTROL_ENABLED`). Its purpose is to run arbitrary code
inside the Scrapy process, so anyone who can connect to it can do that, e.g.
via the Scrapy MCP server.
The server listens on a random localhost port and requires a token for
authentication. This token is stored in a job file (see
`REMOTE_CONTROL_JOBS_DIR` for the location of these files), so you
should protect these files from unauthorized access. On Linux and macOS systems
Scrapy sets file system permissions for job files and the directory containing
them to be accessible only by the owner.
> [!NOTE]
> The server doesn’t use HTTPS so it’s possible to sniff the traffic or
> tamper with it, but it requires the attacker to get access to the loopback
> traffic.
If you do not use this feature, disable it entirely:
```python
REMOTE_CONTROL_ENABLED = False
```
* **Pro:** removes a local code-execution surface and one less listening port.
* **Con:** you can no longer inspect and control a running crawler through it.
### Telnet console
Scrapy enables the telnet console by default
(`TELNETCONSOLE_ENABLED`). The telnet console is a Python shell
running inside the Scrapy process, so anyone who can connect to it can run
arbitrary code in that process.
By default the console binds to `127.0.0.1` (`TELNETCONSOLE_HOST`)
and is protected by a username (`TELNETCONSOLE_USERNAME`, default
`scrapy`) and an automatically generated password
(`TELNETCONSOLE_PASSWORD`), so it is only reachable from the local
machine.
> [!WARNING]
> Telnet does not provide any transport-layer security, so the
> username/password authentication does not protect the credentials or the
> session from anyone able to observe the traffic. Never expose the telnet
> console over an untrusted network by changing `TELNETCONSOLE_HOST`
> to a non-local address.
If you do not use the telnet console, disable it entirely:
```python
TELNETCONSOLE_ENABLED = False
```
* **Pro:** removes a local code-execution surface and one less listening port.
* **Con:** you can no longer inspect and control a running crawler through it.
### Credential leakage across domains
Some Scrapy features attach credentials or other sensitive headers to requests,
and a crawl that spans multiple domains can leak them to unintended hosts:
* HTTP authentication credentials set through
`HttpAuthMiddleware` are only
sent to the domain set in `HTTPAUTH_DOMAIN`. Leave this set to the
intended domain rather than `None` so that credentials are not sent to
every domain you crawl.
* The `Referer` header may disclose the URLs you crawl to other sites. The
default `REFERRER_POLICY` already avoids sending the referrer from
HTTPS to HTTP, but you can tighten it further (for example, to
`same-origin` or `no-referrer`) if needed.
### Job state integrity
When you use `JOBDIR` to pause and resume crawls, the default scheduler disk queues
(`SCHEDULER_DISK_QUEUE`, `SCHEDULER_START_DISK_QUEUE`) write
requests to a compact binary format that is only guaranteed to be complete
after a clean shutdown. If the process is killed, runs out of disk space, or
shares its job directory with a second process, those files can be left
truncated, and resuming the crawl then either fails with an unrecoverable
error or silently drops the pending requests.
To store the scheduler queues in an SQLite database instead, where each
request is written within its own transaction:
```python
SCHEDULER_DISK_QUEUE = "scrapy.squeues.PickleLifoSQLiteQueue"
SCHEDULER_START_DISK_QUEUE = "scrapy.squeues.PickleFifoSQLiteQueue"
```
* **Pro:** an interrupted write cannot corrupt the queues or discard pending
requests unnoticed.
* **Con:** scheduling requests becomes slower, which can be noticeable in
crawls with a high request throughput.
Source: /topics/optimize.md
## Optimization
A crawl goes as fast as its slowest part allows. Find out which part that
is before changing any setting.
Broad crawls have their own set of recommended
adjustments.
### Finding the bottleneck
The bottleneck depends on the spider: on the same machine, one crawl can be
limited by its own parsing code and another by the target website. So measure
the crawl that you want to optimize.
`LogStats` reports crawl speed every
`LOGSTATS_INTERVAL` seconds:
```text
[scrapy.extensions.logstats] INFO: Crawled 1200 pages (at 60 pages/min), scraped 1150 items (at 58 items/min)
```
A rate that stays flat as you raise `CONCURRENT_REQUESTS` means
something else is the limit.
#### Reading the engine status
The telnet console reports, through `est()`,
what every part of the engine is doing at a given moment:
```text
len(engine.downloader.active) : 16
len(engine.scheduler.mqs) : 92
len(engine.scraper.slot.active) : 0
engine.scraper.slot.active_size : 0
engine.scraper.slot.needs_backout() : False
```
A coding agent using the Scrapy MCP server can also
retrieve these values directly.
Take a few readings at different points of the crawl:
- `len(engine.downloader.active)` stays at `CONCURRENT_REQUESTS`:
the downloader is the limit. You are waiting on the network or on the
target website. See optimize-concurrency.
- `len(engine.downloader.active)` stays below
`CONCURRENT_REQUESTS` while the scheduler queues (`mqs`,
`dqs`) hold requests: something throttles those requests before they
reach the downloader, usually `CONCURRENT_REQUESTS_PER_DOMAIN`,
`DOWNLOAD_DELAY` or AutoThrottle.
- Both the downloader and the scheduler queues stay near empty: your spider
is not producing requests fast enough. A crawl that walks pagination one
page at a time cannot use more concurrency than it creates. See
optimize-requests.
- `needs_backout()` is `True`, or `active_size` approaches
`SCRAPER_SLOT_MAX_ACTIVE_SIZE`: responses arrive faster than your
callbacks and item pipelines handle them. The
bottleneck is your own code.
- `len(engine.scheduler.mqs)` grows without settling: the crawl discovers
requests faster than it downloads them. This is what makes long crawls run
out of memory.
#### Reading resource usage
CPU
: Scrapy runs in a single process, and everything except DNS resolution and
code you explicitly move to a thread runs in a single thread. One CPU core
is the ceiling; a process sitting at 100% of a core is CPU-bound no matter
how many cores the machine has.
<br/>
Use a sampling profiler, such as [py-spy](https://github.com/benfred/py-spy), to find out which code is
spending that CPU. Selectors and item pipelines
are the usual answer.
Memory
: The memory usage extension records
`memusage/startup` and `memusage/max`. A `memusage/max`
far above `memusage/startup` is expected; what matters is whether it
keeps growing for as long as the crawl runs.
<br/>
Growth that tracks `len(engine.scheduler.mqs)` is a scheduling problem,
covered in optimize-memory. Growth that does not is a memory
leak.
Network
: Compare `downloader/response_bytes` over the crawl time against your
available bandwidth. Saturated bandwidth caps concurrency regardless of any
setting.
<br/>
DNS resolution is separate: it runs on a thread pool of
`REACTOR_THREADPOOL_MAXSIZE` threads, and results are cached
(`DNSCACHE_ENABLED`, `DNSCACHE_SIZE`). It only becomes a
limit of its own when there are many different domains to resolve, as in
broad crawls, where it shows up as slow starts and
DNS timeouts.
Disk
: Feed exports write to disk on most crawls,
although item data is usually small enough for that not to matter. The ones
to suspect are
`HttpCacheMiddleware` and
the media pipelines, which write whole
responses, and `JOBDIR`, which writes every scheduled request.
### Sending more requests at a time
`CONCURRENT_REQUESTS` caps how many requests are being downloaded at
any given moment, `CONCURRENT_REQUESTS_PER_DOMAIN` caps how many of
those may target the same domain, and `DOWNLOAD_DELAY` sets a minimum
wait between two consecutive requests to the same domain. A project generated by
`startproject` gets one request per second per domain out of these.
Raise them to crawl a single website faster, and see
broad-crawls-concurrency to spread requests across many websites
instead.
The limit that matters, though, is the one the target website tolerates.
Exceeding it gets you throttled, served errors or banned, all of which make the
crawl slower than a lower concurrency would have been. To find that limit:
- Read the robots.txt file of the website. Scrapy
does not act on its `Crawl-delay` and `Request-rate` directives, so when
they are present, translate them into `DOWNLOAD_DELAY` and
concurrency settings yourself.
- Check the traffic that the website already gets, using a service like
[SimilarWeb](https://www.similarweb.com/) or [Cloudflare Radar](https://radar.cloudflare.com/). A rate that is a rounding error next
to what the website serves anyway is unlikely to be a problem for it.
- Look for a documented way in. An API, a bulk export or a search endpoint is
both faster for you and cheaper for the website than crawling its pages, and
the terms of service may state a rate.
- Crawl when the website is idle, in its own timezone, so that the capacity
you take is capacity nobody else wanted.
- Raise concurrency gradually and watch the website respond.
`downloader/response_status_count/{status_code}` counts for 429, 503
or the ban page of the website, growing `retry/count`, or a
download latency that climbs as you push harder,
all mean you have gone past the limit.
### Producing requests faster
A spider that discovers its requests one response at a time keeps the
downloader idle no matter how high you set `CONCURRENT_REQUESTS`. To
put more requests in the scheduler earlier:
- Request every page at once when you can work out how many there are, e.g.
from a page count or from a result count and a page size in the first
response, instead of following a link to the next page on every response.
- Get URLs from a source that lists many of them at once, such as a sitemap
or a search or export endpoint of the target website. For a crawl that
needs nothing else, `SitemapSpider` reads sitemaps
for you.
- Raise the `priority` of pagination requests, so that
they are downloaded before the requests that they compete with, and
discover the rest of the crawl sooner.
Each of these trades memory for speed: a request produced before the downloader
can take it waits in the scheduler, or on disk if you set `JOBDIR`.
Pushed far enough, they turn memory or disk into your new bottleneck, which is
why optimize-memory recommends the reverse of the last point.
### Lowering resource usage
#### Lowering memory usage
- Lower `SCRAPER_SLOT_MAX_ACTIVE_SIZE`.
- Lower `DOWNLOAD_MAXSIZE`, which allows a single response to take up
to 1 GiB of memory by default, multiplied by your concurrency. Set
`DOWNLOAD_WARNSIZE` first to find out whether the website actually
serves responses that big.
- Lower the number of scheduled requests held in
memory:
- Increase the `priority` of requests whose
`callback` cannot yield additional requests.
For example, the following spider uses a higher priority (1) for book
requests than for pagination requests:
```python
from scrapy import Spider
class BooksToScrapeComSpider(Spider):
name = "books_toscrape_com"
start_urls = [
"http://books.toscrape.com/catalogue/category/books/mystery_3/index.html"
]
def parse(self, response):
next_page_links = response.css(".next a")
yield from response.follow_all(next_page_links)
book_links = response.css("article a")
yield from response.follow_all(book_links, callback=self.parse_book, priority=1)
def parse_book(self, response):
yield {
"name": response.css("h1::text").get(),
"price": response.css(".price_color::text").re_first("£(.*)"),
"url": response.url,
}
```
> [!NOTE]
> If the number of request-yielding, low-priority requests
> scheduled at any given time is lower than concurrency settings
> (`CONCURRENT_REQUESTS_PER_DOMAIN` or
> `CONCURRENT_REQUESTS`), as in the example above, this can
> slow down your crawl by turning those requests into a bottleneck.
- If you have many start requests, consider
delaying their iteration.
- Set `JOBDIR` to offload all scheduled requests to disk.
- Be on the lookout for memory leaks.
#### Lowering network usage
- Enable `HttpCacheMiddleware`
while developing your spider, so that re-runs do not download the same
responses again.
#### Lowering CPU usage
- Set `LOG_LEVEL` to `"INFO"` or higher.
- Restrict what you parse. A selector over a
smaller part of the response, or a single query whose result you reuse,
beats repeated queries over the whole document.
#### Other tips
- Try using the asyncio reactor with [uvloop](https://github.com/MagicStack/uvloop) as
custom event loop, i.e. setting
`ASYNCIO_EVENT_LOOP` to `"uvloop.Loop"`.
Alternatively, try switching to a non-asyncio reactor.
- Disable unused components.
For example, set `COOKIES_ENABLED` to `False` unless you need
cookies.
- Split the crawl across separate processes to use more than one CPU core.
See distributed-crawls.
### Speeding up broad crawls
While Scrapy is well suited for **broad crawls**, i.e. crawls that target many
websites, the default settings are optimized for
crawls targeting a single website.
For broad crawls, consider these adjustments:
- Increase the global concurrency:
- Set `CONCURRENT_REQUESTS` as close to
`CONCURRENT_REQUESTS_PER_DOMAIN` × [number of target domains]
(e.g. 8 × 10 domains = 80 concurrent requests) as your CPU and memory
allow.
- Increase `SCRAPER_SLOT_MAX_ACTIVE_SIZE` when increasing
`CONCURRENT_REQUESTS` stops making a difference.
- If memory is a bottleneck, see if crawling in BFO order lowers
memory usage.
- Improve DNS resolution speed:
- Set up your own DNS server, with a local cache and upstream to a [large
DNS server](https://en.wikipedia.org/wiki/Public_recursive_name_server#Notable_public_DNS_service_operators), to avoid slowing down your network.
- Increase `REACTOR_THREADPOOL_MAXSIZE` to the minimum value
that avoids DNS resolution timeouts and makes a noticeable positive
impact in crawl speed.
- Lower the negative impact of some responses:
- Set `RETRY_ENABLED` to `False` or, if you need retries,
consider lowering `RETRY_TIMES`.
- Lower `DOWNLOAD_TIMEOUT` to a more reasonable value, to
discard stuck requests more quickly.
- Set `REDIRECT_ENABLED` to `False` unless you want to follow
redirects.
Source: /topics/developer-tools.md
## Using your browser’s Developer Tools for scraping
Here is a general guide on how to use your browser’s Developer Tools
to ease the scraping process. Today almost all browsers come with
built in [Developer Tools](https://en.wikipedia.org/wiki/Web_development_tools) and although we will use Firefox in this
guide, the concepts are applicable to any other browser.
In this guide we’ll introduce the basic tools to use from a browser’s
Developer Tools by scraping [quotes.toscrape.com](https://quotes.toscrape.com).
### Caveats with inspecting the live browser DOM
Since Developer Tools operate on a live browser DOM, what you’ll actually see
when inspecting the page source is not the original HTML, but a modified one
after applying some browser clean up and executing JavaScript code. Firefox,
in particular, is known for adding `<tbody>` elements to tables. Scrapy, on
the other hand, does not modify the original page HTML, so you won’t be able to
extract any data if you use `<tbody>` in your XPath expressions.
Therefore, you should keep in mind the following things:
* Disable JavaScript while inspecting the DOM looking for XPaths to be
used in Scrapy (in the Developer Tools settings click Disable JavaScript)
* Never use full XPath paths, use relative and clever ones based on attributes
(such as `id`, `class`, `width`, etc) or any identifying features like
`contains(@href, 'image')`.
* Never include `<tbody>` elements in your XPath expressions unless you
really know what you’re doing
### Inspecting a website
By far the most handy feature of the Developer Tools is the Inspector
feature, which allows you to inspect the underlying HTML code of
any webpage. To demonstrate the Inspector, let’s look at the
[quotes.toscrape.com](https://quotes.toscrape.com)-site.
On the site we have a total of ten quotes from various authors with specific
tags, as well as the Top Ten Tags. Let’s say we want to extract all the quotes
on this page, without any meta-information about authors, tags, etc.
Instead of viewing the whole source code for the page, we can simply right click
on a quote and select `Inspect Element (Q)`, which opens up the Inspector.
In it you should see something like this:
The interesting part for us is this:
```html
<div class="quote" itemscope="" itemtype="http://schema.org/CreativeWork">
<span class="text" itemprop="text">(...)</span>
<span>(...)</span>
<div class="tags">(...)</div>
</div>
```
If you hover over the first `div` directly above the `span` tag highlighted
in the screenshot, you’ll see that the corresponding section of the webpage gets
highlighted as well. So now we have a section, but we can’t find our quote text
anywhere.
The advantage of the Inspector is that it automatically expands and collapses
sections and tags of a webpage, which greatly improves readability. You can
expand and collapse a tag by clicking on the arrow in front of it or by double
clicking directly on the tag. If we expand the `span` tag with the `class=
"text"` we will see the quote-text we clicked on. The Inspector lets you
copy XPaths to selected elements. Let’s try it out.
First open the Scrapy shell at [https://quotes.toscrape.com/](https://quotes.toscrape.com/) in a terminal:
```none
$ scrapy shell "https://quotes.toscrape.com/"
```
Then, back to your web browser, right-click on the `span` tag, select
`Copy > XPath` and paste it in the Scrapy shell like so:
```pycon
>>> response.xpath("/html/body/div/div[2]/div[1]/div[1]/span[1]/text()").getall()
['“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”']
```
Adding `text()` at the end we are able to extract the first quote with this
basic selector. But this XPath is not really that clever. All it does is
go down a desired path in the source code starting from `html`. So let’s
see if we can refine our XPath a bit:
If we check the Inspector again we’ll see that directly beneath our
expanded `div` tag we have nine identical `div` tags, each with the
same attributes as our first. If we expand any of them, we’ll see the same
structure as with our first quote: Two `span` tags and one `div` tag. We can
expand each `span` tag with the `class="text"` inside our `div` tags and
see each quote:
```html
<div class="quote" itemscope="" itemtype="http://schema.org/CreativeWork">
<span class="text" itemprop="text">
“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”
</span>
<span>(...)</span>
<div class="tags">(...)</div>
</div>
```
With this knowledge we can refine our XPath: Instead of a path to follow,
we’ll simply select all `span` tags with the `class="text"` by using
the [has-class-extension](https://parsel.readthedocs.io/en/latest/usage.html#other-xpath-extensions):
```pycon
>>> response.xpath('//span[has-class("text")]/text()').getall()
['“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”',
'“It is our choices, Harry, that show what we truly are, far more than our abilities.”',
'“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”',
...]
```
And with one simple, cleverer XPath we are able to extract all quotes from
the page. We could have constructed a loop over our first XPath to increase
the number of the last `div`, but this would have been unnecessarily
complex and by simply constructing an XPath with `has-class("text")`
we were able to extract all quotes in one line.
The Inspector has a lot of other helpful features, such as searching in the
source code or directly scrolling to an element you selected. Let’s demonstrate
a use case:
Say you want to find the `Next` button on the page. Type `Next` into the
search bar on the top right of the Inspector. You should get two results.
The first is a `li` tag with the `class="next"`, the second the text
of an `a` tag. Right click on the `a` tag and select `Scroll into View`.
If you hover over the tag, you’ll see the button highlighted. From here
we could easily create a Link Extractor to
follow the pagination. On a simple site such as this, there may not be
the need to find an element visually but the `Scroll into View` function
can be quite useful on complex sites.
Note that the search bar can also be used to search for and test CSS
selectors. For example, you could search for `span.text` to find
all quote texts. Instead of a full text search, this searches for
exactly the `span` tag with the `class="text"` in the page.
### The Network-tool
While scraping you may come across dynamic webpages where some parts
of the page are loaded dynamically through multiple requests. While
this can be quite tricky, the Network-tool in the Developer Tools
greatly facilitates this task. To demonstrate the Network-tool, let’s
take a look at the page [quotes.toscrape.com/scroll](https://quotes.toscrape.com/scroll).
The page is quite similar to the basic [quotes.toscrape.com](https://quotes.toscrape.com)-page,
but instead of the above-mentioned `Next` button, the page
automatically loads new quotes when you scroll to the bottom. We
could go ahead and try out different XPaths directly, but instead
we’ll check another quite useful command from the Scrapy shell:
```none
$ scrapy shell "quotes.toscrape.com/scroll"
(...)
>>> view(response)
```
A browser window should open with the webpage but with one
crucial difference: Instead of the quotes we just see a greenish
bar with the word `Loading...`.
The `view(response)` command let’s us view the response our
shell or later our spider receives from the server. Here we see
that some basic template is loaded which includes the title,
the login-button and the footer, but the quotes are missing. This
tells us that the quotes are being loaded from a different request
than `quotes.toscrape/scroll`.
If you click on the `Network` tab, you will probably only see
two entries. The first thing we do is enable persistent logs by
clicking on `Persist Logs`. If this option is disabled, the
log is automatically cleared each time you navigate to a different
page. Enabling this option is a good default, since it gives us
control on when to clear the logs.
If we reload the page now, you’ll see the log get populated with six
new requests.
Here we see every request that has been made when reloading the page
and can inspect each request and its response. So let’s find out
where our quotes are coming from:
First click on the request with the name `scroll`. On the right
you can now inspect the request. In `Headers` you’ll find details
about the request headers, such as the URL, the method, the IP-address,
and so on. We’ll ignore the other tabs and click directly on `Response`.
What you should see in the `Preview` pane is the rendered HTML-code,
that is exactly what we saw when we called `view(response)` in the
shell. Accordingly the `type` of the request in the log is `html`.
The other requests have types like `css` or `js`, but what
interests us is the one request called `quotes?page=1` with the
type `json`.
If we click on this request, we see that the request URL is
`https://quotes.toscrape.com/api/quotes?page=1` and the response
is a JSON-object that contains our quotes. We can also right-click
on the request and open `Open in new tab` to get a better overview.
With this response we can now easily parse the JSON-object and
also request each page to get every quote on the site:
```python
import scrapy
class QuoteSpider(scrapy.Spider):
name = "quote"
allowed_domains = ["quotes.toscrape.com"]
page = 1
start_urls = ["https://quotes.toscrape.com/api/quotes?page=1"]
def parse(self, response):
data = response.json()
for quote in data["quotes"]:
yield {"quote": quote["text"]}
if data["has_next"]:
self.page += 1
url = f"https://quotes.toscrape.com/api/quotes?page={self.page}"
yield scrapy.Request(url=url, callback=self.parse)
```
This spider starts at the first page of the quotes-API. With each
response, we parse the `response.text` and assign it to `data`.
This lets us operate on the JSON-object like on a Python dictionary.
We iterate through the `quotes` and print out the `quote["text"]`.
If the handy `has_next` element is `true` (try loading
[quotes.toscrape.com/api/quotes?page=10](https://quotes.toscrape.com/api/quotes?page=10) in your browser or a
page-number greater than 10), we increment the `page` attribute
and `yield` a new request, inserting the incremented page-number
into our `url`.
In more complex websites, it could be difficult to easily reproduce the
requests, as we could need to add `headers` or `cookies` to make it work.
In those cases you can export the requests in [cURL](https://curl.se/)
format, by right-clicking on each of them in the network tool and using the
`from_curl()` method to generate an equivalent
request:
```python
from scrapy import Request
request = Request.from_curl(
"curl 'https://quotes.toscrape.com/api/quotes?page=1' -H 'User-Agent: Mozil"
"la/5.0 (X11; Linux x86_64; rv:67.0) Gecko/20100101 Firefox/67.0' -H 'Acce"
"pt: */*' -H 'Accept-Language: ca,en-US;q=0.7,en;q=0.3' --compressed -H 'X"
"-Requested-With: XMLHttpRequest' -H 'Proxy-Authorization: Basic QFRLLTAzM"
"zEwZTAxLTk5MWUtNDFiNC1iZWRmLTJjNGI4M2ZiNDBmNDpAVEstMDMzMTBlMDEtOTkxZS00MW"
"I0LWJlZGYtMmM0YjgzZmI0MGY0' -H 'Connection: keep-alive' -H 'Referer: http"
"://quotes.toscrape.com/scroll' -H 'Cache-Control: max-age=0'"
)
```
Alternatively, if you want to know the arguments needed to recreate that
request you can use the `curl_to_request_kwargs()`
function to get a dictionary with the equivalent arguments:
#### scrapy.utils.curl.curl_to_request_kwargs(curl_command: str, ignore_unknown_options: bool = True) → dict[str, Any]
Convert a cURL command syntax to Request kwargs.
* **Parameters:**
* **curl_command** (*str*) – string containing the curl command
* **ignore_unknown_options** (*bool*) – If true, only a warning is emitted when
cURL options are unknown. Otherwise
raises an error. (default: True)
* **Returns:**
dictionary of Request kwargs
Note that to translate a cURL command into a Scrapy request,
you may use [curl2scrapy](https://michael-shub.github.io/curl2scrapy/).
As you can see, with a few inspections in the Network-tool we
were able to easily replicate the dynamic requests of the scrolling
functionality of the page. Crawling dynamic pages can be quite
daunting and pages can be very complex, but it (mostly) boils down
to identifying the correct request and replicating it in your spider.
Source: /topics/dynamic-content.md
## Selecting dynamically-loaded content
Some webpages show the desired data when you load them in a web browser.
However, when you download them using Scrapy, you cannot reach the desired data
using selectors.
When this happens, the recommended approach is to
find the data source and extract the data
from it.
If you fail to do that, and you can nonetheless access the desired data through
the DOM from your web browser, see
topics-headless-browsing.
### Finding the data source
To extract the desired data, you must first find its source location.
If the data is in a non-text-based format, such as an image or a PDF document,
use the network tool of your web browser to find
the corresponding request, and reproduce it.
If your web browser lets you select the desired data as text, the data may be
defined in embedded JavaScript code, or loaded from an external resource in a
text-based format.
In that case, you can use a tool like [wgrep](https://github.com/stav/wgrep) to find the URL of that resource.
If the data turns out to come from the original URL itself, you must
inspect the source code of the webpage to
determine where the data is located.
If the data comes from a different URL, you will need to reproduce the
corresponding request.
### Inspecting the source code of a webpage
Sometimes you need to inspect the source code of a webpage (not the
DOM) to determine where some desired data is located.
Use Scrapy’s `fetch` command to download the webpage contents as seen
by Scrapy:
```
scrapy fetch --nolog https://example.com > response.html
```
If the desired data is in embedded JavaScript code within a `<script/>`
element, see topics-parsing-javascript.
If you cannot find the desired data, first make sure it’s not just Scrapy:
download the webpage with an HTTP client like [curl](https://curl.se/) or [wget](https://www.gnu.org/software/wget/) and see if the
information can be found in the response they get.
If they get a response with the desired data, modify your Scrapy
`Request` to match that of the other HTTP client. For
example, try using the same user-agent string (`USER_AGENT`) or the
same `headers`.
If they also get a response without the desired data, you’ll need to take
steps to make your request more similar to that of the web browser. See
topics-reproducing-requests.
### Reproducing requests
Sometimes we need to reproduce a request the way our web browser performs it.
Use the network tool of your web browser to see
how your web browser performs the desired request, and try to reproduce that
request with Scrapy.
It might be enough to yield a `Request` with the same HTTP
method and URL. However, you may also need to reproduce the body, headers and
form parameters (see form) of that request.
As all major browsers allow to export the requests in [curl](https://curl.se/) format, Scrapy
incorporates the method `from_curl()` to generate an equivalent
`Request` from a cURL command. To get more information
visit request from curl inside the network
tool section.
Once you get the expected response, you can extract the desired data from
it.
You can reproduce any request with Scrapy. However, some times reproducing all
necessary requests may not seem efficient in developer time. If that is your
case, and crawling speed is not a major concern for you, you can alternatively
consider using a headless browser.
If you get the expected response sometimes, but not always, the issue is
probably not your request, but the target server. The target server might be
buggy, overloaded, or banning some of your requests.
Note that to translate a cURL command into a Scrapy request,
you may use [curl2scrapy](https://michael-shub.github.io/curl2scrapy/).
### Handling different response formats
Once you have a response with the desired data, how you extract the desired
data from it depends on the type of response:
- If the response is HTML, XML or JSON, use selectors as usual.
- If the response is JSON, use `response.json()` to load the desired data:
```python
data = response.json()
```
If the desired data is inside HTML or XML code embedded within JSON data,
you can load that HTML or XML code into a
`Selector` and then
use it as usual:
```python
selector = Selector(text=data["html"])
```
- If the response is JavaScript, or HTML with a `<script/>` element
containing the desired data, see topics-parsing-javascript.
- If the response is CSS, use a regular expression to
extract the desired data from
`response.text`.
- If the response is an image or another format based on images (e.g. PDF),
read the response as bytes from
`response.body` and use an OCR
solution to extract the desired data as text.
For example, you can use [pytesseract](https://github.com/madmaze/pytesseract). To read a table from a PDF,
[tabula-py](https://github.com/chezou/tabula-py) may be a better choice.
- If the response is SVG, or HTML with embedded SVG containing the desired
data, you may be able to extract the desired data using
selectors, since SVG is based on XML.
Otherwise, you might need to convert the SVG code into a raster image, and
handle that raster image.
### Parsing JavaScript code
If the desired data is hardcoded in JavaScript, you first need to get the
JavaScript code:
- If the JavaScript code is in a JavaScript file, simply read
`response.text`.
- If the JavaScript code is within a `<script/>` element of an HTML page,
use selectors to extract the text within that
`<script/>` element.
Once you have a string with the JavaScript code, you can extract the desired
data from it:
- You might be able to use a regular expression to
extract the desired data in JSON format, which you can then parse with
`json.loads()`.
For example, if the JavaScript code contains a separate line like
`var data = {"field": "value"};` you can extract that data as follows:
```pycon
>>> pattern = r"\bvar\s+data\s*=\s*(\{.*?\})\s*;\s*\n"
>>> json_data = response.css("script::text").re_first(pattern)
>>> json.loads(json_data)
{'field': 'value'}
```
- [chompjs](https://github.com/Nykakin/chompjs) provides an API to parse JavaScript objects into a `dict`.
For example, if the JavaScript code contains
`var data = {field: "value", secondField: "second value"};`
you can extract that data as follows:
```pycon
>>> import chompjs
>>> javascript = response.css("script::text").get()
>>> data = chompjs.parse_js_object(javascript)
>>> data
{'field': 'value', 'secondField': 'second value'}
```
- Otherwise, use [js2xml](https://github.com/scrapinghub/js2xml) to convert the JavaScript code into an XML document
that you can parse using selectors.
For example, if the JavaScript code contains
`var data = {field: "value"};` you can extract that data as follows:
```pycon
>>> import js2xml
>>> import lxml.etree
>>> from parsel import Selector
>>> javascript = response.css("script::text").get()
>>> xml = lxml.etree.tostring(js2xml.parse(javascript), encoding="unicode")
>>> selector = Selector(text=xml)
>>> selector.css('var[name="data"]').get()
'<var name="data"><object><property name="field"><string>value</string></property></object></var>'
```
### Using a headless browser
On webpages that fetch data from additional requests, reproducing those
requests that contain the desired data is the preferred approach. The effort is
often worth the result: structured, complete data with minimum parsing time and
network transfer.
However, sometimes it can be really hard to reproduce certain requests. Or you
may need something that no request can give you, such as a screenshot of a
webpage as seen in a web browser. In this case using a [headless browser](https://en.wikipedia.org/wiki/Headless_browser) will
help.
A headless browser is a special web browser that provides an API for
automation. By installing the asyncio reactor,
it is possible to integrate `asyncio`-based libraries which handle headless browsers.
One such library is [playwright-python](https://github.com/microsoft/playwright-python) (an official Python port of [playwright](https://github.com/microsoft/playwright)).
The following is a simple snippet to illustrate its usage within a Scrapy spider:
```python
import scrapy
from playwright.async_api import async_playwright
class PlaywrightSpider(scrapy.Spider):
name = "playwright"
start_urls = ["data:,"] # avoid using the default Scrapy downloader
async def parse(self, response):
async with async_playwright() as pw:
browser = await pw.chromium.launch()
page = await browser.new_page()
await page.goto("https://example.org")
title = await page.title()
return {"title": title}
```
However, using [playwright-python](https://github.com/microsoft/playwright-python) directly as in the above example
circumvents most of the Scrapy components (middlewares, dupefilter, etc).
We recommend using [scrapy-playwright](https://github.com/scrapy-plugins/scrapy-playwright) for a better integration.
Source: /topics/leaks.md
## Debugging memory leaks
In Scrapy, objects such as requests, responses and items have a finite
lifetime: they are created, used for a while, and finally destroyed.
From all those objects, the Request is probably the one with the longest
lifetime, as it stays waiting in the Scheduler queue until it’s time to process
it. For more info see topics-architecture.
As these Scrapy objects have a (rather long) lifetime, there is always the risk
of accumulating them in memory without releasing them properly and thus causing
what is known as a “memory leak”.
To help debugging memory leaks, Scrapy provides a built-in mechanism for
tracking objects references called trackref,
and you can also use a third-party library called muppy for more advanced memory debugging (see below for more
info). Both mechanisms must be used after connecting to a live crawl.
### Common causes of memory leaks
It happens quite often (sometimes by accident, sometimes on purpose) that the
Scrapy developer passes objects referenced in Requests (for example, using the
`cb_kwargs` or `meta`
attributes or the request callback function) and that effectively bounds the
lifetime of those referenced objects to the lifetime of the Request. This is,
by far, the most common cause of memory leaks in Scrapy projects, and a quite
difficult one to debug for newcomers.
In big projects, the spiders are typically written by different people and some
of those spiders could be “leaking” and thus affecting the rest of the other
(well-written) spiders when they get to run concurrently, which, in turn,
affects the whole crawling process.
The leak could also come from a custom middleware, pipeline or extension that
you have written, if you are not releasing the (previously allocated) resources
properly. For example, allocating resources on `spider_opened`
but not releasing them on `spider_closed` may cause problems if
you’re running multiple spiders per process.
#### Too Many Requests?
By default Scrapy keeps the request queue in memory; it includes
`Request` objects and all objects
referenced in Request attributes (e.g. in `cb_kwargs`
and `meta`).
While not necessarily a leak, this can take a lot of memory. Enabling
persistent job queue could help keeping memory usage
in control.
### Debugging memory leaks with `trackref`
`scrapy.utils.trackref` is a module provided by Scrapy to debug the most
common cases of memory leaks. It basically tracks the references to all live
Request, Response, Item, Spider and Selector objects.
You can enter the telnet console and inspect how many objects (of the classes
mentioned above) are currently alive using the `prefs()` function which is an
alias to the `print_live_refs()` function:
```bash
telnet localhost 6023
```
```pycon
>>> prefs()
Live References
ExampleSpider 1 oldest: 15s ago
HtmlResponse 10 oldest: 1s ago
Selector 2 oldest: 0s ago
Request 878 oldest: 7s ago
```
As you can see, that report also shows the “age” of the oldest object in each
class. If you’re running multiple spiders per process chances are you can
figure out which spider is leaking by looking at the oldest request or response.
You can get the oldest object of each class using the
`get_oldest()` function (from the telnet console).
#### Which objects are tracked?
The objects tracked by `trackref` are all from these classes (and all its
subclasses):
* `scrapy.Request`
* `scrapy.http.Response`
* `scrapy.Item`
* `scrapy.Selector`
* `scrapy.Spider`
#### A real example
Let’s see a concrete example of a hypothetical case of memory leaks.
Suppose we have some spider with a line similar to this one:
```python
return Request(
f"http://www.somenastyspider.com/product.php?pid={product_id}",
callback=self.parse,
cb_kwargs={"referer": response},
)
```
That line is passing a response reference inside a request which effectively
ties the response lifetime to the requests’ one, and that would definitely
cause memory leaks.
Let’s see how we can discover the cause (without knowing it
a priori, of course) by using the `trackref` tool.
After the crawler is running for a few minutes and we notice its memory usage
has grown a lot, we can enter its telnet console and check the live
references:
```pycon
>>> prefs()
Live References
SomenastySpider 1 oldest: 15s ago
HtmlResponse 3890 oldest: 265s ago
Selector 2 oldest: 0s ago
Request 3878 oldest: 250s ago
```
The fact that there are so many live responses (and that they’re so old) is
definitely suspicious, as responses should have a relatively short lifetime
compared to Requests. The number of responses is similar to the number
of requests, so it looks like they are tied in a some way. We can now go
and check the code of the spider to discover the nasty line that is
generating the leaks (passing response references inside requests).
Sometimes extra information about live objects can be helpful.
Let’s check the oldest response:
```pycon
>>> from scrapy.utils.trackref import get_oldest
>>> r = get_oldest("HtmlResponse")
>>> r.url
'http://www.somenastyspider.com/product.php?pid=123'
```
If you want to iterate over all objects, instead of getting the oldest one, you
can use the `scrapy.utils.trackref.iter_all()` function:
```pycon
>>> from scrapy.utils.trackref import iter_all
>>> [r.url for r in iter_all("HtmlResponse")]
['http://www.somenastyspider.com/product.php?pid=123',
'http://www.somenastyspider.com/product.php?pid=584',
...]
```
#### Too many spiders?
If your project has too many spiders executed in parallel,
the output of `prefs()` can be difficult to read.
For this reason, that function has a `ignore` argument which can be used to
ignore a particular class (and all its subclasses). For
example, this won’t show any live references to spiders:
```pycon
>>> from scrapy.spiders import Spider
>>> prefs(ignore=Spider)
```
#### scrapy.utils.trackref module
Here are the functions available in the `trackref` module.
#### *class* scrapy.utils.trackref.object_ref(\*args: Any, \*\*kwargs: Any)
Inherit from this class if you want to track live instances with the
`trackref` module.
#### scrapy.utils.trackref.print_live_refs(ignore=NoneType)
Print a report of live references, grouped by class name.
* **Parameters:**
**ignore** (*type* *or* *tuple*) – if given, all objects from the specified class (or tuple of
classes) will be ignored.
#### scrapy.utils.trackref.get_oldest(class_name: str) → Any
Return the oldest object alive with the given class name, or `None` if
none is found. Use `print_live_refs()` first to get a list of all
tracked live objects per class name.
#### scrapy.utils.trackref.iter_all(class_name: str) → Iterable[Any]
Return an iterator over all objects alive with the given class name. Use
`print_live_refs()` first to get a list of all tracked live objects per
class name.
### Debugging memory leaks with muppy
`trackref` provides a very convenient mechanism for tracking down memory
leaks, but it only keeps track of the objects that are more likely to cause
memory leaks. However, there are other cases where the memory leaks could come
from other (more or less obscure) objects. If this is your case, and you can’t
find your leaks using `trackref`, you still have another resource: the muppy
library.
You can use muppy from [Pympler](https://pypi.org/project/Pympler/).
If you use `pip`, you can install muppy with the following command:
```
pip install Pympler
```
Here’s an example to view all Python objects available in
the heap using muppy:
```pycon
>>> from pympler import muppy
>>> all_objects = muppy.get_objects()
>>> len(all_objects)
28667
>>> from pympler import summary
>>> suml = summary.summarize(all_objects)
>>> summary.print_(suml)
types | # objects | total size
==================================== | =========== | ============
<class 'str | 9822 | 1.10 MB
<class 'dict | 1658 | 856.62 KB
<class 'type | 436 | 443.60 KB
<class 'code | 2974 | 419.56 KB
<class '_io.BufferedWriter | 2 | 256.34 KB
<class 'set | 420 | 159.88 KB
<class '_io.BufferedReader | 1 | 128.17 KB
<class 'wrapper_descriptor | 1130 | 88.28 KB
<class 'tuple | 1304 | 86.57 KB
<class 'weakref | 1013 | 79.14 KB
<class 'builtin_function_or_method | 958 | 67.36 KB
<class 'method_descriptor | 865 | 60.82 KB
<class 'abc.ABCMeta | 62 | 59.96 KB
<class 'list | 446 | 58.52 KB
<class 'int | 1425 | 43.20 KB
```
For more info about muppy, refer to the [muppy documentation](https://pythonhosted.org/Pympler/muppy.html).
### Leaks without leaks
Sometimes, you may notice that the memory usage of your Scrapy process will
only increase, but never decrease. Unfortunately, this could happen even
though neither Scrapy nor your project are leaking memory. This is due to a
(not so well) known problem of Python, which may not return released memory to
the operating system in some cases. For more information on this issue see:
* [Python Memory Management](https://www.evanjones.ca/python-memory.html)
* [Python Memory Management Part 2](https://www.evanjones.ca/python-memory-part2.html)
* [Python Memory Management Part 3](https://www.evanjones.ca/python-memory-part3.html)
The improvements proposed by Evan Jones, which are detailed in [this paper](https://www.evanjones.ca/memoryallocator/),
got merged in Python 2.5, but this only reduces the problem, it doesn’t fix it
completely. To quote the paper:
> *Unfortunately, this patch can only free an arena if there are no more
> objects allocated in it anymore. This means that fragmentation is a large
> issue. An application could have many megabytes of free memory, scattered
> throughout all the arenas, but it will be unable to free any of it. This is
> a problem experienced by all memory allocators. The only way to solve it is
> to move to a compacting garbage collector, which is able to move objects in
> memory. This would require significant changes to the Python interpreter.*
To keep memory consumption reasonable you can split the job into several
smaller jobs or enable persistent job queue
and stop/start spider from time to time.
Source: /topics/media-pipeline.md
## Downloading and processing files and images
Scrapy provides reusable item pipelines for
downloading files attached to a particular item (for example, when you scrape
products and also want to download their images locally). These pipelines share
a bit of functionality and structure (we refer to them as media pipelines), but
typically you’ll either use the Files Pipeline or the Images Pipeline.
Both pipelines implement these features:
* Avoid re-downloading media that was downloaded recently
* Specifying where to store the media (filesystem directory, FTP server, Amazon S3 bucket,
Google Cloud Storage bucket)
The Images Pipeline has a few extra functions for processing images:
* Convert all downloaded images to a common format (JPG) and mode (RGB)
* Thumbnail generation
* Check images width/height to make sure they meet a minimum constraint
The pipelines also keep an internal queue of those media URLs which are currently
being scheduled for download, and connect those responses that arrive containing
the same media to that queue. This avoids downloading the same media more than
once when it’s shared by several items.
### Using the Files Pipeline
The typical workflow, when using the `FilesPipeline` goes like
this:
1. In a Spider, you scrape an item and put the URLs of the desired into a
`file_urls` field.
2. The item is returned from the spider and goes to the item pipeline.
3. When the item reaches the `FilesPipeline`, the URLs in the
`file_urls` field are downloaded using the standard Scrapy downloader
(which means the downloader middlewares are used, but the spider middlewares
aren’t). The item remains “locked” at that particular pipeline stage until
the files have finished downloading (or failed for some reason).
4. When the files are downloaded, another field (`files`) will be populated
with the results. This field will contain a list of dicts with information
about the downloaded files, such as the downloaded path, the original
scraped url (taken from the `file_urls` field), the file checksum and the file status.
The files in the list of the `files` field will retain the same order of
the original `file_urls` field. If some file failed downloading, an
error will be logged and the file won’t be present in the `files` field.
### Using the Images Pipeline
> [!NOTE]
> Requires the images extra.
Using the `ImagesPipeline` is a lot like using the `FilesPipeline`,
except the default field names used are different: you use `image_urls` for
the image URLs of an item and it will populate an `images` field for the information
about the downloaded images.
The advantage of using the `ImagesPipeline` for image files is that you
can configure some extra functions like generating thumbnails and filtering
the images based on their size.
### Enabling your Media Pipeline
To enable your media pipeline you must first add it to your project
`ITEM_PIPELINES` setting.
For Images Pipeline, use:
```python
ITEM_PIPELINES = {"scrapy.pipelines.images.ImagesPipeline": 1}
```
For Files Pipeline, use:
```python
ITEM_PIPELINES = {"scrapy.pipelines.files.FilesPipeline": 1}
```
> [!NOTE]
> You can also use both the Files and Images Pipeline at the same time.
Then, configure the target storage setting to a valid value that will be used
for storing the downloaded images. Otherwise the pipeline will remain disabled,
even if you include it in the `ITEM_PIPELINES` setting.
For the Files Pipeline, set the `FILES_STORE` setting:
```python
FILES_STORE = "/path/to/valid/dir"
```
For the Images Pipeline, set the `IMAGES_STORE` setting:
```python
IMAGES_STORE = "/path/to/valid/dir"
```
### File Naming
#### Default File Naming
By default, files are stored using an [SHA-1 hash](https://en.wikipedia.org/wiki/SHA_hash_functions) of their URLs for the file names.
For example, the following image URL:
```
http://www.example.com/image.jpg
```
Whose `SHA-1 hash` is:
```
3afec3b4765f8f0a07b78f98c07b83f013567a0a
```
Will be downloaded and stored using your chosen storage method and the following file name:
```
3afec3b4765f8f0a07b78f98c07b83f013567a0a.jpg
```
#### Custom File Naming
You may wish to use a different calculated file name for saved files.
For example, classifying an image by including meta in the file name.
Customize file names by overriding the `file_path` method of your
media pipeline.
For example, an image pipeline with image URL:
```
http://www.example.com/product/images/large/front/0000000004166
```
Can be processed into a file name with a condensed hash and the perspective
`front`:
```
00b08510e4_front.jpg
```
By overriding `file_path` like this:
```python
import hashlib
def file_path(self, request, response=None, info=None, *, item=None):
image_url_hash = hashlib.shake_256(request.url.encode()).hexdigest(5)
image_perspective = request.url.split("/")[-2]
image_filename = f"{image_url_hash}_{image_perspective}.jpg"
return image_filename
```
> [!WARNING]
> If your custom file name scheme relies on meta data that can vary between
> scrapes it may lead to unexpected re-downloading of existing media using
> new file names.
>
> For example, if your custom file name scheme uses a product title and the
> site changes an item’s product title between scrapes, Scrapy will re-download
> the same media using updated file names.
For more information about the `file_path` method, see topics-media-pipeline-override.
#### Naming files after the response
`file_path` also receives the `response`, which allows naming files after
response data. For example, to determine the file extension from the
`Content-Type` header, for URLs that do not end in a file name:
```python
import mimetypes
from scrapy.pipelines.files import FilesPipeline
class ContentTypeFilesPipeline(FilesPipeline):
def file_path(self, request, response=None, info=None, *, item=None):
path = super().file_path(request, response, info, item=item)
if response is None:
return path
content_type = response.headers["Content-Type"].decode()
return path + (mimetypes.guess_extension(content_type) or "")
```
This requires setting `FILES_EXPIRES` to `0`. To find out whether a
file has already been downloaded, Scrapy calls `file_path` before the
download, with `response` set to `None`, and checks the age of the file at
the resulting path. A path that depends on the response can never match that
check, and `FILES_EXPIRES` set to `0` disables it, at the cost of
downloading every file on every run.
#### Setting request headers
Media requests are built from the item, not from the response that yielded
it, so features like `RefererMiddleware`
do not apply to them. To send a header such as `Referer`, store the value
on the item in your spider, and read it back from the item in
`get_media_requests` of your custom media pipeline:
```python
from scrapy import Request, Spider
from scrapy.pipelines.files import FilesPipeline
class MySpider(Spider):
def parse(self, response):
yield {
"file_urls": response.css("a.file::attr(href)").getall(),
"referrer": response.url,
}
class MyFilesPipeline(FilesPipeline):
def get_media_requests(self, item, info):
for file_url in item["file_urls"]:
yield Request(file_url, headers={"Referer": item["referrer"]})
```
### Supported Storage
#### File system storage
File system storage will save files to the following path:
```
<IMAGES_STORE>/full/<FILE_NAME>
```
Where:
* `<IMAGES_STORE>` is the directory defined in `IMAGES_STORE` setting
for the Images Pipeline.
* `full` is a sub-directory to separate full images from thumbnails (if
used). For more info see topics-images-thumbnails.
* `<FILE_NAME>` is the file name assigned to the file. For more info see topics-file-naming.
#### FTP server storage
`FILES_STORE` and `IMAGES_STORE` can point to an FTP server.
Scrapy will automatically upload the files to the server.
`FILES_STORE` and `IMAGES_STORE` should be written in one of the
following forms:
```
ftp://username:password@address:port/path
ftp://address:port/path
```
If `username` and `password` are not provided, they are taken from the `FTP_USER` and
`FTP_PASSWORD` settings respectively.
FTP supports two different connection modes: active or passive. Scrapy uses
the passive connection mode by default. To use the active connection mode instead,
set the `FEED_STORAGE_FTP_ACTIVE` setting to `True`.
#### Amazon S3 storage
> [!NOTE]
> Requires the s3 extra.
`FILES_STORE` and `IMAGES_STORE` can represent an Amazon S3
bucket. Scrapy will automatically upload the files to the bucket.
For example, this is a valid `IMAGES_STORE` value:
```python
IMAGES_STORE = "s3://bucket/images"
```
You can modify the Access Control List (ACL) policy used for the stored files,
which is defined by the `FILES_STORE_S3_ACL` and
`IMAGES_STORE_S3_ACL` settings. By default, the ACL is set to
`private`. To make the files publicly available use the `public-read`
policy:
```python
IMAGES_STORE_S3_ACL = "public-read"
```
For more information, see [canned ACLs](https://docs.aws.amazon.com/AmazonS3/latest/userguide/acl-overview.html#canned-acl) in the Amazon S3 Developer Guide.
You can also use other S3-like storages. Storages like self-hosted [Minio](https://github.com/minio/minio) or
[Zenko CloudServer](https://www.zenko.io/cloudserver/). All you need to do is set endpoint option in you Scrapy
settings:
```python
AWS_ENDPOINT_URL = "http://minio.example.com:9000"
```
For self-hosting you also might feel the need not to use SSL and not to verify SSL connection:
```python
AWS_USE_SSL = False # or True (None by default)
AWS_VERIFY = False # or True (None by default)
```
To reuse connections for as many files as you check or upload in parallel, set
`AWS_MAX_POOL_CONNECTIONS` accordingly.
#### Google Cloud Storage
> [!NOTE]
> Requires the gcs extra.
`FILES_STORE` and `IMAGES_STORE` can represent a Google Cloud
Storage bucket. Scrapy will automatically upload the files to the bucket.
For example, these are valid `IMAGES_STORE` and `GCS_PROJECT_ID` settings:
```python
IMAGES_STORE = "gs://bucket/images/"
GCS_PROJECT_ID = "project_id"
```
For information about authentication, see this [documentation](https://docs.cloud.google.com/docs/authentication).
You can modify the Access Control List (ACL) policy used for the stored files,
which is defined by the `FILES_STORE_GCS_ACL` and
`IMAGES_STORE_GCS_ACL` settings. By default, the ACL is set to
`''` (empty string) which means that Cloud Storage applies the bucket’s default object ACL to the object.
To make the files publicly available use the `publicRead`
policy:
```python
IMAGES_STORE_GCS_ACL = "publicRead"
```
For more information, see [Predefined ACLs](https://docs.cloud.google.com/storage/docs/access-control/lists#predefined-acl) in the Google Cloud Platform Developer Guide.
### Usage example
In order to use a media pipeline, first enable it.
Then, if a spider returns an item object with the URLs
field (`file_urls` or `image_urls`, for the Files or Images Pipeline
respectively), the pipeline will put the results under the respective field
(`files` or `images`).
When using item types for which fields are defined beforehand,
you must define both the URLs field and the results field. For example, when
using the images pipeline, items must define both the `image_urls` and the
`images` field. For instance, using a dataclass:
```python
from dataclasses import dataclass, field
@dataclass
class MyItem:
# ... other item fields ...
image_urls: list[str] = field(default_factory=list)
images: list[dict] = field(default_factory=list)
```
If you want to use another field name for the URLs key or for the results key,
it is also possible to override it.
For the Files Pipeline, set `FILES_URLS_FIELD` and/or
`FILES_RESULT_FIELD` settings:
```python
FILES_URLS_FIELD = "field_name_for_your_files_urls"
FILES_RESULT_FIELD = "field_name_for_your_processed_files"
```
For the Images Pipeline, set `IMAGES_URLS_FIELD` and/or
`IMAGES_RESULT_FIELD` settings:
```python
IMAGES_URLS_FIELD = "field_name_for_your_images_urls"
IMAGES_RESULT_FIELD = "field_name_for_your_processed_images"
```
If you need something more complex and want to override the custom pipeline
behaviour, see topics-media-pipeline-override.
If you have multiple image pipelines inheriting from `ImagesPipeline`
and you want to have different settings in different pipelines you can set
setting keys preceded with uppercase name of your pipeline class. E.g. if your
pipeline is called `MyPipeline` and you want to have custom
`IMAGES_URLS_FIELD` you define setting
`MYPIPELINE_IMAGES_URLS_FIELD` and your custom settings will be used.
### Additional features
#### File expiration
The Image Pipeline avoids downloading files that were downloaded recently. To
adjust this retention delay use the `FILES_EXPIRES` setting (or
`IMAGES_EXPIRES`, in case of Images Pipeline), which
specifies the delay in number of days:
```python
# 120 days of delay for files expiration
FILES_EXPIRES = 120
# 30 days of delay for images expiration
IMAGES_EXPIRES = 30
```
The default value for both settings is 90 days.
If you have pipeline that subclasses FilesPipeline and you’d like to have
different setting for it you can set setting keys preceded by uppercase
class name. E.g. given pipeline class called MyPipeline you can set setting key:
> MYPIPELINE_FILES_EXPIRES = 180
and pipeline class MyPipeline will have expiration time set to 180.
The last modified time from the file is used to determine the age of the file in days,
which is then compared to the set expiration time to determine if the file is expired.
#### Thumbnail generation for images
The Images Pipeline can automatically create thumbnails of the downloaded
images.
In order to use this feature, you must set `IMAGES_THUMBS` to a dictionary
where the keys are the thumbnail names and the values are their dimensions.
For example:
```python
IMAGES_THUMBS = {
"small": (50, 50),
"big": (270, 270),
}
```
When you use this feature, the Images Pipeline will create thumbnails of the
each specified size with this format:
```
<IMAGES_STORE>/thumbs/<size_name>/<image_id>.jpg
```
Where:
* `<size_name>` is the one specified in the `IMAGES_THUMBS`
dictionary keys (`small`, `big`, etc)
* `<image_id>` is the [SHA-1 hash](https://en.wikipedia.org/wiki/SHA_hash_functions) of the image url
Example of image files stored using `small` and `big` thumbnail names:
```
<IMAGES_STORE>/full/63bbfea82b8880ed33cdb762aa11fab722a90a24.jpg
<IMAGES_STORE>/thumbs/small/63bbfea82b8880ed33cdb762aa11fab722a90a24.jpg
<IMAGES_STORE>/thumbs/big/63bbfea82b8880ed33cdb762aa11fab722a90a24.jpg
```
The first one is the full image, as downloaded from the site.
#### Filtering out small images
When using the Images Pipeline, you can drop images which are too small, by
specifying the minimum allowed size in the `IMAGES_MIN_HEIGHT` and
`IMAGES_MIN_WIDTH` settings.
For example:
```python
IMAGES_MIN_HEIGHT = 110
IMAGES_MIN_WIDTH = 110
```
> [!NOTE]
> The size constraints don’t affect thumbnail generation at all.
It is possible to set just one size constraint or both. When setting both of
them, only images that satisfy both minimum sizes will be saved. For the
above example, images of sizes (105 x 105) or (105 x 200) or (200 x 105) will
all be dropped because at least one dimension is shorter than the constraint.
By default, there are no size constraints, so all images are processed.
#### Allowing redirections
By default media pipelines ignore redirects, i.e. an HTTP redirection
to a media file URL request will mean the media download is considered failed.
To handle media redirections, set this setting to `True`:
```python
MEDIA_ALLOW_REDIRECTS = True
```
### Extending the Media Pipelines
See here the methods that you can override in your custom Files Pipeline:
#### *class* scrapy.pipelines.files.FilesPipeline
##### file_path(self, request, response=None, info=None, \*, item=None)
This method is called once per downloaded item. It returns the
download path of the file originating from the specified
`response`.
In addition to `response`, this method receives the original
`request`,
`info` and
`item`
You can override this method to customize the download path of each file.
For example, if file URLs end like regular paths (e.g.
`https://example.com/a/b/c/foo.png`), you can use the following
approach to download all files into the `files` folder with their
original filenames (e.g. `files/foo.png`):
```python
from pathlib import PurePosixPath
from scrapy.utils.httpobj import urlparse_cached
from scrapy.pipelines.files import FilesPipeline
class MyFilesPipeline(FilesPipeline):
def file_path(self, request, response=None, info=None, *, item=None):
return "files/" + PurePosixPath(urlparse_cached(request).path).name
```
Similarly, you can use the `item` to determine the file path based on some item
property, or the `response`, see file-naming-response.
By default the `file_path()` method returns
`full/<request URL hash>.<extension>`.
##### get_media_requests(item, info)
As seen on the workflow, the pipeline will get the requests for the files
to download from the item by calling this method. You can override it to
change what requests are returned:
```python
from itemadapter import ItemAdapter
def get_media_requests(self, item, info):
adapter = ItemAdapter(item)
for file_url in adapter["file_urls"]:
yield scrapy.Request(file_url)
```
You can also use it to set request headers, see media-request-headers.
Those requests will be processed by the pipeline and, when they have finished
downloading, the results will be sent to the
`item_completed()` method, as a list of 2-element tuples.
Each tuple will contain `(success, file_info_or_error)` where:
* `success` is a boolean which is `True` if the image was downloaded
successfully or `False` if it failed for some reason
* `file_info_or_error` is a dict containing the following keys (if
success is `True`) or a `Failure` if
there was a problem.
* `url` - the url where the file was downloaded from. This is the url of
the request returned from the `get_media_requests()`
method.
* `path` - the path (relative to `FILES_STORE`) where the file
was stored
* `checksum` - a [MD5 hash](https://en.wikipedia.org/wiki/MD5) of the image contents
* `status` - the file status indication.
It can be one of the following:
* `downloaded` - file was downloaded.
* `uptodate` - file was not downloaded, as it was downloaded recently,
according to the file expiration policy.
* `cached` - file was taken from a cache (the response has a
`"cached"` flag, e.g. from
`HttpCacheMiddleware`).
The list of tuples received by `item_completed()` is
guaranteed to retain the same order of the requests returned from the
`get_media_requests()` method.
Here’s a typical value of the `results` argument:
```python
[
(
True,
{
"checksum": "2b00042f7481c7b056c4b410d28f33cf",
"path": "full/0a79c461a4062ac383dc4fade7bc09f1384a3910.jpg",
"url": "http://www.example.com/files/product1.pdf",
"status": "downloaded",
},
),
(False, Failure(...)),
]
```
##### item_completed(results, item, info)
The `FilesPipeline.item_completed()` method called when all file
requests for a single item have completed (either finished downloading, or
failed for some reason).
The `item_completed()` method must return the
output that will be sent to subsequent item pipeline stages, so you must
return (or drop) the item, as you would in any pipeline.
Here is an example of the `item_completed()` method where we
store the downloaded file paths (passed in results) in the `file_paths`
item field, and we drop the item if it doesn’t contain any files:
```python
from itemadapter import ItemAdapter
from scrapy.exceptions import DropItem
def item_completed(self, results, item, info):
file_paths = [x["path"] for ok, x in results if ok]
if not file_paths:
raise DropItem("Item contains no files")
adapter = ItemAdapter(item)
adapter["file_paths"] = file_paths
return item
```
By default, the `item_completed()` method returns the item.
See here the methods that you can override in your custom Images Pipeline:
#### *class* scrapy.pipelines.images.ImagesPipeline
> The `ImagesPipeline` is an extension of the `FilesPipeline`,
> customizing the field names and adding custom behavior for images.
##### file_path(self, request, response=None, info=None, \*, item=None)
This method is called once per downloaded item. It returns the
download path of the file originating from the specified
`response`.
In addition to `response`, this method receives the original
`request`,
`info` and
`item`
You can override this method to customize the download path of each file.
For example, if file URLs end like regular paths (e.g.
`https://example.com/a/b/c/foo.png`), you can use the following
approach to download all files into the `files` folder with their
original filenames (e.g. `files/foo.png`):
```python
from pathlib import PurePosixPath
from scrapy.utils.httpobj import urlparse_cached
from scrapy.pipelines.images import ImagesPipeline
class MyImagesPipeline(ImagesPipeline):
def file_path(self, request, response=None, info=None, *, item=None):
return "files/" + PurePosixPath(urlparse_cached(request).path).name
```
Similarly, you can use the `item` to determine the file path based on some item
property, or the `response`, see file-naming-response.
By default the `file_path()` method returns
`full/<request URL hash>.<extension>`.
##### thumb_path(self, request, thumb_id, response=None, info=None, \*, item=None)
This method is called for every item of `IMAGES_THUMBS` per downloaded item. It returns the
thumbnail download path of the image originating from the specified
`response`.
In addition to `response`, this method receives the original
`request`,
`thumb_id`,
`info` and
`item`.
You can override this method to customize the thumbnail download path of each image.
You can use the `item` to determine the file path based on some item
property.
By default the `thumb_path()` method returns
`thumbs/<size name>/<request URL hash>.<extension>`.
##### get_media_requests(item, info)
Works the same way as `FilesPipeline.get_media_requests()` method,
but using a different field name for image urls.
Must return a Request for each image URL.
##### item_completed(results, item, info)
The `ImagesPipeline.item_completed()` method is called when all image
requests for a single item have completed (either finished downloading, or
failed for some reason).
Works the same way as `FilesPipeline.item_completed()` method,
but using a different field names for storing image downloading results.
By default, the `item_completed()` method returns the item.
### Accessing the response of a failed download
When a download fails, `item_completed()` receives it as a
`Failure` whose `value` is the exception
raised from `media_downloaded()`. For a non-2xx response, that
exception is a `FileException` built from just
a message, without a reference to the response. To access the response, for
example to record its `status` on the item, override
`media_downloaded()` to attach it to a custom exception:
```python
from scrapy.pipelines.files import FilesPipeline
from scrapy.pipelines.media import FileException
class ResponseFileException(FileException):
def __init__(self, response):
super().__init__(f"download-error ({response.status})")
self.response = response
class MyFilesPipeline(FilesPipeline):
async def media_downloaded(self, response, request, info, *, item=None):
try:
return await super().media_downloaded(response, request, info, item=item)
except FileException:
raise ResponseFileException(response)
def item_completed(self, results, item, info):
errors = [
value.value.response.status
for ok, value in results
if not ok and value.check(ResponseFileException)
]
if errors:
item["download_errors"] = errors
return super().item_completed(results, item, info)
```
### Custom Images pipeline example
Here is a full example of the Images Pipeline whose methods are exemplified
above:
```python
import scrapy
from itemadapter import ItemAdapter
from scrapy.exceptions import DropItem
from scrapy.pipelines.images import ImagesPipeline
class MyImagesPipeline(ImagesPipeline):
def get_media_requests(self, item, info):
for image_url in item["image_urls"]:
yield scrapy.Request(image_url)
def item_completed(self, results, item, info):
image_paths = [x["path"] for ok, x in results if ok]
if not image_paths:
raise DropItem("Item contains no images")
adapter = ItemAdapter(item)
adapter["image_paths"] = image_paths
return item
```
To enable your custom media pipeline component you must add its class import path to the
`ITEM_PIPELINES` setting, like in the following example:
```python
ITEM_PIPELINES = {"myproject.pipelines.MyImagesPipeline": 300}
```
#### Content-based image filtering pipeline
This example overrides `get_images()` to filter images using a classifier,
such as a [TensorFlow](https://tensorflow.org) model. Override `is_valid_image()` with your
classification logic:
```python
from scrapy.pipelines.images import ImagesPipeline, ImageException
class ImageClassifierPipeline(ImagesPipeline):
def is_valid_image(self, image):
raise NotImplementedError
def get_images(self, response, request, info, *, item=None):
for path, image, buf in super().get_images(response, request, info, item=item):
if not self.is_valid_image(image):
raise ImageException("Image does not match criteria")
yield path, image, buf
```
Source: /topics/deploy.md
## Deploying Spiders
This section describes the different options you have for deploying your Scrapy
spiders to run them on a regular basis. Running Scrapy spiders in your local
machine is very convenient for the (early) development stage, but not so much
when you need to execute long-running spiders or move spiders to run in
production continuously. This is where the solutions for deploying Scrapy
spiders come in.
Popular choices for deploying Scrapy spiders are:
* Scrapyd (open source)
* Zyte Scrapy Cloud (cloud-based)
### Deploying to a Scrapyd Server
[Scrapyd](https://github.com/scrapy/scrapyd) is an open source application to run Scrapy spiders. It provides
a server with HTTP API, capable of running and monitoring Scrapy spiders.
To deploy spiders to Scrapyd, you can use the scrapyd-deploy tool provided by
the [scrapyd-client](https://github.com/scrapy/scrapyd-client) package. Please refer to the [scrapyd-deploy
documentation](https://scrapyd.readthedocs.io/en/latest/deploy.html) for more information.
Scrapyd is maintained by some of the Scrapy developers.
### Deploying to Zyte Scrapy Cloud
[Zyte Scrapy Cloud](https://www.zyte.com/scrapy-cloud/) is a hosted, cloud-based service by [Zyte](https://www.zyte.com/), the company
behind Scrapy.
Zyte Scrapy Cloud removes the need to setup and monitor servers and provides a
nice UI to manage spiders and review scraped items, logs and stats.
To deploy spiders to Zyte Scrapy Cloud you can use the [shub](https://shub.readthedocs.io/en/latest/) command line
tool.
Please refer to the [Zyte Scrapy Cloud documentation](https://docs.zyte.com/scrapy-cloud.md) for more information.
Zyte Scrapy Cloud is compatible with Scrapyd and one can switch between
them as needed - the configuration is read from the `scrapy.cfg` file
just like `scrapyd-deploy`.
Source: /topics/autothrottle.md
## AutoThrottle extension
This is an extension for automatically throttling crawling speed based on load
of both the Scrapy server and the website you are crawling.
### Design goals
1. be nicer to sites instead of using default download delay of zero
2. automatically adjust Scrapy to the optimum crawling speed, so the user
doesn’t have to tune the download delays to find the optimum one.
The user only needs to specify the maximum concurrent requests
it allows, and the extension does the rest.
### How it works
Scrapy allows defining the concurrency and delay of different download slots,
e.g. through the `DOWNLOAD_SLOTS` setting. By default requests are
assigned to slots based on their URL domain, although it is possible to
customize the download slot of any request.
The AutoThrottle extension adjusts the delay of each download slot dynamically,
to make your spider send `AUTOTHROTTLE_TARGET_CONCURRENCY` concurrent
requests on average to each remote website.
It uses download latency to compute the delays. The main idea is the
following: if a server needs `latency` seconds to respond, a client
should send a request each `latency/N` seconds to have `N` requests
processed in parallel.
Instead of adjusting the delays one can just set a small fixed
download delay and impose hard limits on concurrency using
`CONCURRENT_REQUESTS_PER_DOMAIN`. It will provide a similar
effect, but there are some important differences:
* because the download delay is small there will be occasional bursts
of requests;
* often non-200 (error) responses can be returned faster than regular
responses, so with a small download delay and a hard concurrency limit
crawler will be sending requests to server faster when server starts to
return errors. But this is an opposite of what crawler should do - in case
of errors it makes more sense to slow down: these errors may be caused by
the high request rate.
AutoThrottle doesn’t have these issues.
### Throttling algorithm
AutoThrottle algorithm adjusts download delays based on the following rules:
1. spiders always start with a download delay of
`AUTOTHROTTLE_START_DELAY`;
2. when a response is received, the target download delay is calculated as
`latency / N` where `latency` is a latency of the response,
and `N` is `AUTOTHROTTLE_TARGET_CONCURRENCY`.
3. download delay for next requests is set to the average of previous
download delay and the target download delay;
4. latencies of non-200 responses are not allowed to decrease the delay;
5. download delay can’t become less than `DOWNLOAD_DELAY` or greater
than `AUTOTHROTTLE_MAX_DELAY`
> [!NOTE]
> The AutoThrottle extension honours the standard Scrapy settings for
> concurrency and delay. This means that it will respect
> `CONCURRENT_REQUESTS_PER_DOMAIN` and
> never set a download delay lower than `DOWNLOAD_DELAY`.
In Scrapy, the download latency is measured as the time elapsed between
sending the request and receiving the HTTP headers.
Note that these latencies are very hard to measure accurately in a cooperative
multitasking environment because Scrapy may be busy processing a spider
callback, for example, and unable to attend downloads. However, these latencies
should still give a reasonable estimate of how busy Scrapy (and ultimately, the
server) is, and this extension builds on that premise.
### Prevent specific requests from triggering slot delay adjustments
> [!WARNING]
> Added in version 2.12.0.
AutoThrottle adjusts the delay of download slots based on the latencies of
responses that belong to that download slot. The only exceptions are non-200
responses, which are only taken into account to increase that delay, but
ignored if they would decrease that delay.
You can also set the `autothrottle_dont_adjust_delay` request metadata key to
`True` in any request to prevent its response latency from impacting the
delay of its download slot:
```python
from scrapy import Request
Request("https://example.com", meta={"autothrottle_dont_adjust_delay": True})
```
Note, however, that AutoThrottle still determines the starting delay of every
download slot. If you want AutoThrottle not to impact a download slot at all,
in addition to setting this meta key in all requests that use that download
slot, you might want to set a custom value for the `delay` attribute of that
download slot, e.g. using `DOWNLOAD_SLOTS`.
### Settings
The settings used to control the AutoThrottle extension are:
* `AUTOTHROTTLE_ENABLED`
* `AUTOTHROTTLE_START_DELAY`
* `AUTOTHROTTLE_MAX_DELAY`
* `AUTOTHROTTLE_TARGET_CONCURRENCY`
* `AUTOTHROTTLE_DEBUG`
* `CONCURRENT_REQUESTS_PER_DOMAIN`
* `DOWNLOAD_DELAY`
For more information see autothrottle-algorithm.
#### AUTOTHROTTLE_ENABLED
Default: `False`
Enables the AutoThrottle extension.
#### AUTOTHROTTLE_START_DELAY
Default: `5.0`
The initial download delay (in seconds).
#### AUTOTHROTTLE_MAX_DELAY
Default: `60.0`
The maximum download delay (in seconds) to be set in case of high latencies.
#### AUTOTHROTTLE_TARGET_CONCURRENCY
Default: `1.0`
Average number of requests Scrapy should be sending in parallel to remote
websites. It must be higher than `0.0`.
By default, AutoThrottle adjusts the delay to send a single
concurrent request to each of the remote websites. Set this option to
a higher value (e.g. `2.0`) to increase the throughput and the load on remote
servers. A lower `AUTOTHROTTLE_TARGET_CONCURRENCY` value
(e.g. `0.5`) makes the crawler more conservative and polite.
Note that `CONCURRENT_REQUESTS_PER_DOMAIN` is still respected
when AutoThrottle extension is enabled. This means that if
`AUTOTHROTTLE_TARGET_CONCURRENCY` is set to a value higher than
`CONCURRENT_REQUESTS_PER_DOMAIN`, the crawler won’t reach this number
of concurrent requests.
At every given time point Scrapy can be sending more or less concurrent
requests than `AUTOTHROTTLE_TARGET_CONCURRENCY`; it is a suggested
value the crawler tries to approach, not a hard limit.
#### AUTOTHROTTLE_DEBUG
Default: `False`
Enable AutoThrottle debug mode which will display stats on every response
received, so you can see how the throttling parameters are being adjusted in
real time.
Source: /topics/benchmarking.md
## Benchmarking
Scrapy comes with a simple benchmarking suite that spawns a local HTTP server
and crawls it at the maximum possible speed. The goal of this benchmarking is
to get an idea of how Scrapy performs in your hardware, in order to have a
common baseline for comparisons. It uses a simple spider that does nothing and
just follows links.
To run it use:
```
scrapy bench
```
You should see an output like this:
```
2016-12-16 21:18:48 [scrapy.utils.log] INFO: Scrapy 1.2.2 started (bot: quotesbot)
2016-12-16 21:18:48 [scrapy.utils.log] INFO: Overridden settings: {'CLOSESPIDER_TIMEOUT': 10, 'ROBOTSTXT_OBEY': True, 'SPIDER_MODULES': ['quotesbot.spiders'], 'LOGSTATS_INTERVAL': 1, 'BOT_NAME': 'quotesbot', 'LOG_LEVEL': 'INFO', 'NEWSPIDER_MODULE': 'quotesbot.spiders'}
2016-12-16 21:18:49 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.closespider.CloseSpider',
'scrapy.extensions.logstats.LogStats',
'scrapy.extensions.telnet.TelnetConsole',
'scrapy.extensions.corestats.CoreStats']
2016-12-16 21:18:49 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.offsite.OffsiteMiddleware',
'scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
'scrapy.downloadermiddlewares.retry.RetryMiddleware',
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
2016-12-16 21:18:49 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
'scrapy.spidermiddlewares.referer.RefererMiddleware',
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
'scrapy.spidermiddlewares.depth.DepthMiddleware']
2016-12-16 21:18:49 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2016-12-16 21:18:49 [scrapy.core.engine] INFO: Spider opened
2016-12-16 21:18:49 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2016-12-16 21:18:50 [scrapy.extensions.logstats] INFO: Crawled 70 pages (at 4200 pages/min), scraped 0 items (at 0 items/min)
2016-12-16 21:18:51 [scrapy.extensions.logstats] INFO: Crawled 134 pages (at 3840 pages/min), scraped 0 items (at 0 items/min)
2016-12-16 21:18:52 [scrapy.extensions.logstats] INFO: Crawled 198 pages (at 3840 pages/min), scraped 0 items (at 0 items/min)
2016-12-16 21:18:53 [scrapy.extensions.logstats] INFO: Crawled 254 pages (at 3360 pages/min), scraped 0 items (at 0 items/min)
2016-12-16 21:18:54 [scrapy.extensions.logstats] INFO: Crawled 302 pages (at 2880 pages/min), scraped 0 items (at 0 items/min)
2016-12-16 21:18:55 [scrapy.extensions.logstats] INFO: Crawled 358 pages (at 3360 pages/min), scraped 0 items (at 0 items/min)
2016-12-16 21:18:56 [scrapy.extensions.logstats] INFO: Crawled 406 pages (at 2880 pages/min), scraped 0 items (at 0 items/min)
2016-12-16 21:18:57 [scrapy.extensions.logstats] INFO: Crawled 438 pages (at 1920 pages/min), scraped 0 items (at 0 items/min)
2016-12-16 21:18:58 [scrapy.extensions.logstats] INFO: Crawled 470 pages (at 1920 pages/min), scraped 0 items (at 0 items/min)
2016-12-16 21:18:59 [scrapy.core.engine] INFO: Closing spider (closespider_timeout)
2016-12-16 21:18:59 [scrapy.extensions.logstats] INFO: Crawled 518 pages (at 2880 pages/min), scraped 0 items (at 0 items/min)
2016-12-16 21:19:00 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 229995,
'downloader/request_count': 534,
'downloader/request_method_count/GET': 534,
'downloader/response_bytes': 1565504,
'downloader/response_count': 534,
'downloader/response_status_count/200': 534,
'finish_reason': 'closespider_timeout',
'finish_time': datetime.datetime(2016, 12, 16, 16, 19, 0, 647725),
'log_count/INFO': 17,
'request_depth_max': 19,
'response_received_count': 534,
'scheduler/dequeued': 533,
'scheduler/dequeued/memory': 533,
'scheduler/enqueued': 10661,
'scheduler/enqueued/memory': 10661,
'start_time': datetime.datetime(2016, 12, 16, 16, 18, 49, 799869)}
2016-12-16 21:19:00 [scrapy.core.engine] INFO: Spider closed (closespider_timeout)
```
That tells you that Scrapy is able to crawl about 3000 pages per minute in the
hardware where you run it. Note that this is a very simple spider intended to
follow links, any custom spider you write will probably do more stuff which
results in slower crawl rates. How slower depends on how much your spider does
and how well it’s written.
Use [scrapy-bench](https://github.com/scrapy/scrapy-bench) for more complex benchmarking.
Source: /topics/jobs.md
## Jobs: pausing and resuming crawls
Sometimes, for big sites, it’s desirable to pause crawls and be able to resume
them later.
Scrapy supports this functionality out of the box by providing the following
facilities:
* a scheduler that persists scheduled requests on disk
* a duplicates filter that persists visited requests on disk
* an extension that keeps some spider state (key/value pairs) persistent
between batches
### Job directory
To enable persistence support, define a *job directory* through the
`JOBDIR` setting.
The job directory will store all required data to keep the state of a *single*
job (i.e. a spider run), so that if stopped cleanly, it can be resumed later.
> [!WARNING]
> This directory must *not* be shared by different spiders, or even
> different jobs of the same spider. See job-dir-spider-name.
> [!WARNING]
> Treat the job directory with the same security care as your
> Scrapy project source code. Do not point `JOBDIR` to a path that
> untrusted parties can write to.
See also job-dir-contents.
### How to use it
To start a spider with persistence support enabled, run it like this:
```
scrapy crawl somespider -s JOBDIR=crawls/somespider-1
```
Then, you can stop the spider safely at any time (by pressing Ctrl-C or sending
a signal), and resume it later by issuing the same command:
```
scrapy crawl somespider -s JOBDIR=crawls/somespider-1
```
### Deriving the job directory from the spider name
One way to keep job directories apart is to build the path in
`update_settings()`, so that a group of spiders shares the
same path except for the spider name:
```python
from pathlib import Path
from scrapy import Spider
class BaseSpider(Spider):
@classmethod
def update_settings(cls, settings):
super().update_settings(settings)
settings.set("JOBDIR", str(Path("crawls", cls.name)), priority="spider")
```
### Keeping persistent state between batches
Sometimes you’ll want to keep some persistent spider state between pause/resume
batches. You can use the `spider.state` attribute for that, which should be a
dict. There’s a built-in extension
that takes care of serializing, storing and loading that attribute from the job
directory, when the spider starts and stops.
Here’s an example of a callback that uses the spider state (other spider code
is omitted for brevity):
```python
def parse_item(self, response):
# parse item here
self.state["items_count"] = self.state.get("items_count", 0) + 1
```
### Persistence gotchas
There are a few things to keep in mind if you want to be able to use the Scrapy
persistence support:
#### Pause limitations
Job pausing and resuming is only supported when the spider is paused by
stopping it cleanly. Forced, sudden or otherwise unclean shutdown can lead to
data corruption in the job directory, which may prevent the spider from
resuming correctly.
#### Scrapy version changes
The contents of a job directory are an implementation detail of the Scrapy
version that wrote them. A job must be resumed with the same Scrapy version
that paused it; after upgrading or downgrading Scrapy, start a new job with a
new job directory.
#### Cookies expiration
Cookies may expire. So, if you don’t resume your spider quickly the requests
scheduled may no longer work. This won’t be an issue if your spider doesn’t rely
on cookies.
#### Request serialization
For persistence to work, `Request` objects must be
serializable with `pickle`, except for the callback and errback values passed to their `__init__`
method, which must be methods of the running `Spider` class.
Requests that cannot be serialized are kept in memory only: they are still
sent, but they are lost when the crawl is paused.
If you wish to log the requests that couldn’t be serialized, you can set the
`SCHEDULER_DEBUG` setting to `True` in the project’s settings page.
It is `False` by default.
> [!NOTE]
> Because requests are serialized with `pickle`, the objects you
> store on a request, such as the values of its
> `cb_kwargs` and `meta`
> dictionaries, are deep-copied when the request is written to and later read
> back from the job directory. As a result, the callback receives a *copy* of
> those objects rather than the original ones, and changes made to the copy are
> not reflected in the original object. Keep this in mind if you rely on
> sharing mutable state through `cb_kwargs` or `meta`.
### Job directory contents
The contents of a job directory depend on the components used during the job.
Components known to write in the job directory include the scheduler and the `SpiderState`
extension. See the reference documentation of the corresponding components for
details.
For example, with default settings, the job directory may look like this:
```none
├── requests.queue
| ├── active.json
| └── {hostname}-{hash}
| └── {priority}{s?}
| ├── q{00000}
| └── info.json
├── requests.seen
└── spider.state
```
Where:
- `Scheduler` creates the `requests.queue/`
directory and the `active.json` file, the latter containing the state
data returned by `DownloaderAwarePriorityQueue.close()` the last time the job
was paused.
- `DownloaderAwarePriorityQueue` creates the
`{hostname}-{hash}` directories.
- `ScrapyPriorityQueue` creates the `{priority}{s?}`
directories.
- `scrapy.squeues.PickleFifoDiskQueue`, a subclass of
`queuelib.FifoDiskQueue` that uses `pickle` to serialize
`dict` representations of `scrapy.Request` objects, creates
the `info.json` and `q{00000}` files.
- `RFPDupeFilter` creates the `requests.seen`
file.
- `SpiderState` creates the
`spider.state` file.
Source: /topics/coroutines.md
## Coroutines
Scrapy supports the coroutine syntax
(i.e. `async def`).
### Supported callables
The following callables may be defined as coroutines using `async def`, and
hence use coroutine syntax (e.g. `await`, `async for`, `async with`):
- The `start()` spider method, which *must* be
defined as an asynchronous generator.
> [!WARNING]
> Added in version 2.13.
- `Request` callbacks, which may
also be defined as asynchronous generators.
- The `process_item()` method of
item pipelines.
- The
`process_request()`,
`process_response()`,
and
`process_exception()`
methods of
downloader middlewares.
- The
`process_spider_output()`
method of spider middlewares, which
*must* be defined as an asynchronous generator except in
universal spider middlewares.
- The `process_start()` method
of spider middlewares, which *must* be
defined as an asynchronous generator.
> [!WARNING]
> Added in version 2.13.
- Signal handlers that support deferreds.
- Methods of download handlers.
> [!WARNING]
> Added in version 2.14.
### Using Deferred-based APIs
In addition to native coroutine APIs Scrapy has some APIs that return a
`Deferred` object or take a user-supplied
function that returns a `Deferred` object. These
APIs are also asynchronous but don’t yet support native `async def` syntax.
In the future we plan to add support for the `async def` syntax to these APIs
or replace them with other APIs where changing the existing ones isn’t
possible.
These APIs have a coroutine-based implementation and a Deferred-based one:
- `scrapy.crawler.Crawler`:
- `crawl_async()` (coroutine-based) and
`crawl()` (Deferred-based): the former
may be inconvenient to use in Deferred-based code so both are available,
this may change in a future Scrapy version.
- `scrapy.crawler.AsyncCrawlerRunner` and its subclass
`scrapy.crawler.AsyncCrawlerProcess` (coroutine-based) and
`scrapy.crawler.CrawlerRunner` and its subclass
`scrapy.crawler.CrawlerProcess` (Deferred-based): the former
doesn’t support non-default reactors and so the latter should be used
with those.
The following user-supplied methods can return
`Deferred` objects (the methods that can also
return coroutines are listed in coroutine-support):
- Custom downloader implementations (see `DOWNLOADER`):
- `fetch()`
- Custom scheduler implementations (see `SCHEDULER`):
- `open()`
- `close()`
- Custom dupefilters (see `DUPEFILTER_CLASS`):
- `open()`
- `close()`
- Custom feed storages (see `FEED_STORAGES`):
- `store()`
- Subclasses of `scrapy.pipelines.media.MediaPipeline`:
- `media_to_download()`
- `item_completed()`
- Custom storages used by subclasses of
`scrapy.pipelines.files.FilesPipeline`:
- `persist_file()`
- `stat_file()`
In most cases you can use these APIs in code that otherwise uses coroutines, by
wrapping a `Deferred` object into a
`Future` object or vice versa. See asyncio-await-dfd for
more information about this.
For example: a custom scheduler needs to define an `open()` method that can
return a `Deferred` object. You can write a
method that works with Deferreds and returns one directly, or you can write a
coroutine and convert it into a function that returns a Deferred with
`deferred_f_from_coro_f()`.
### General usage
There are several use cases for coroutines in Scrapy.
Code that would return Deferreds when written for previous Scrapy versions,
such as downloader middlewares and signal handlers, can be rewritten to be
shorter and cleaner:
```python
from itemadapter import ItemAdapter
class DbPipeline:
def _update_item(self, data, item):
adapter = ItemAdapter(item)
adapter["field"] = data
return item
def process_item(self, item):
adapter = ItemAdapter(item)
dfd = db.get_some_data(adapter["id"])
dfd.addCallback(self._update_item, item)
return dfd
```
becomes:
```python
from itemadapter import ItemAdapter
class DbPipeline:
async def process_item(self, item):
adapter = ItemAdapter(item)
adapter["field"] = await db.get_some_data(adapter["id"])
return item
```
Coroutines may be used to call asynchronous code. This includes other
coroutines, functions that return Deferreds and functions that return
awaitable objects such as `Future`.
This means you can use many useful Python libraries providing such code:
```python
class MySpiderDeferred(Spider):
# ...
async def parse(self, response):
additional_response = await treq.get("https://additional.url")
additional_data = await treq.content(additional_response)
# ... use response and additional_data to yield items and requests
class MySpiderAsyncio(Spider):
# ...
async def parse(self, response):
async with aiohttp.ClientSession() as session:
async with session.get("https://additional.url") as additional_response:
additional_data = await additional_response.text()
# ... use response and additional_data to yield items and requests
```
> [!NOTE]
> Many libraries that use coroutines, such as [aio-libs](https://github.com/aio-libs), require the
> `asyncio` loop and to use them you need to
> enable asyncio support in Scrapy.
> [!NOTE]
> If you want to `await` on Deferreds while using the asyncio reactor,
> you need to wrap them.
Common use cases for asynchronous code include:
* requesting data from websites, databases and other services (in
`start()`, callbacks, pipelines and
middlewares);
* storing data in databases (in pipelines and middlewares);
* delaying the spider initialization until some external event (in the
`spider_opened` handler);
* calling asynchronous Scrapy methods like
`ExecutionEngine.download_async()` (see the
screenshot pipeline example).
### Inline requests
The spider below shows how to send a request and await its response all from
within a spider callback:
```python
from scrapy import Spider, Request
class SingleRequestSpider(Spider):
name = "single"
start_urls = ["https://example.org/product"]
async def parse(self, response, **kwargs):
additional_request = Request("https://example.org/price")
additional_response = await self.crawler.engine.download_async(
additional_request
)
yield {
"h1": response.css("h1").get(),
"price": additional_response.css("#price").get(),
}
```
You can also send multiple requests in parallel:
```python
import asyncio
from scrapy import Spider, Request
class MultipleRequestsSpider(Spider):
name = "multiple"
start_urls = ["https://example.com/product"]
async def parse(self, response, **kwargs):
additional_requests = [
Request("https://example.com/price"),
Request("https://example.com/color"),
]
tasks = []
for r in additional_requests:
task = self.crawler.engine.download_async(r)
tasks.append(task)
responses = await asyncio.gather(*tasks)
yield {
"h1": response.css("h1::text").get(),
"price": responses[0].css(".price::text").get(),
"color": responses[1].css(".color::text").get(),
}
```
Source: /topics/asyncio.md
## asyncio
Scrapy supports `asyncio` natively. New projects created with
`startproject` have asyncio enabled by default, and you can use
`asyncio` and `asyncio`-powered libraries in any coroutine.
The rest of this page covers advanced topics. If you are starting a new project,
no additional setup is needed.
### Configuring the asyncio reactor
New projects generated with `startproject` have the asyncio
reactor configured by default. No manual setup is needed.
The `TWISTED_REACTOR` setting controls which Twisted reactor Scrapy
uses. Its default value is
`'twisted.internet.asyncioreactor.AsyncioSelectorReactor'`, which enables
`asyncio` support.
If you are using `AsyncCrawlerRunner` or
`CrawlerRunner`, you also need to
install the `AsyncioSelectorReactor`
reactor manually. You can do that using
`install_reactor()`:
```python
install_reactor("twisted.internet.asyncioreactor.AsyncioSelectorReactor")
```
### Handling a pre-installed reactor
`twisted.internet.reactor` and some other Twisted imports install the default
Twisted reactor as a side effect. Once a Twisted reactor is installed, it is
not possible to switch to a different reactor at run time.
If you configure the asyncio Twisted reactor and, at
run time, Scrapy complains that a different reactor is already installed,
chances are you have some such imports in your code.
You can usually fix the issue by moving those offending module-level Twisted
imports to the method or function definitions where they are used. For example,
if you have something like:
```python
from twisted.internet import reactor
def my_function():
reactor.callLater(...)
```
Switch to something like:
```python
def my_function():
from twisted.internet import reactor
reactor.callLater(...)
```
Alternatively, you can try to manually install the asyncio reactor, with `install_reactor()`, before
those imports happen.
### Integrating Deferred code and asyncio code
Coroutine functions can await on Deferreds by wrapping them into
`asyncio.Future` objects. Scrapy provides two helpers for this:
#### scrapy.utils.defer.deferred_to_future(d: Deferred[\_T]) → Future[\_T]
Return an `asyncio.Future` object that wraps *d*.
This function requires an installed asyncio reactor or a running asyncio
event loop, see using-asyncio.
In this state you cannot await on `Deferred`
objects from Scrapy callables defined as coroutines, you can only await on `Future` objects. Wrapping
`Deferred` objects into `Future` objects allows you to wait on them:
```python
class MySpider(Spider):
...
async def parse(self, response):
deferred = some_dfd_helper()
result = await deferred_to_future(deferred)
```
> [!WARNING]
> Changed in version 2.14: This function no longer installs an asyncio loop if called before the
> Twisted asyncio reactor is installed. A `RuntimeError` is raised
> in this case.
#### scrapy.utils.defer.maybe_deferred_to_future(d: Deferred[\_T]) → Deferred[\_T] | Future[\_T]
Return *d* as an object that can be awaited from a Scrapy callable
defined as a coroutine.
What you can await in Scrapy callables defined as coroutines depends on the
value of `TWISTED_REACTOR` and `TWISTED_REACTOR_ENABLED`:
- When using the asyncio reactor, or not
using a reactor at all, you can only await
on `asyncio.Future` objects.
- When using a non-asyncio reactor, you can only
await on `Deferred` objects.
If you want to write code that uses `Deferred` objects but works in both
of these states, use this function on all `Deferred` objects:
```python
class MySpider(Spider):
...
async def parse(self, response):
deferred = some_dfd_helper()
result = await maybe_deferred_to_future(deferred)
```
> [!TIP]
> If you don’t need to support reactors other than the default
> `AsyncioSelectorReactor`, you
> can use `deferred_to_future()`, otherwise you
> should use `maybe_deferred_to_future()`.
> [!TIP]
> If you need to use these functions in code that aims to be compatible
> with lower versions of Scrapy that do not provide these functions,
> down to Scrapy 2.0 (earlier versions do not support
> `asyncio`), you can copy the implementation of these functions
> into your own code.
Coroutines and futures can be wrapped into Deferreds (for example, when a
Scrapy API requires passing a Deferred to it) using the following helpers:
#### scrapy.utils.defer.deferred_from_coro(o: Awaitable[\_T]) → Deferred[\_T]
#### scrapy.utils.defer.deferred_from_coro(o: \_T2) → \_T2
Convert a coroutine or other awaitable object into a Deferred,
or return the object as is if it isn’t a coroutine.
#### scrapy.utils.defer.deferred_f_from_coro_f(coro_f: Callable[\_P, Awaitable[\_T]]) → Callable[\_P, Deferred[\_T]]
Convert a coroutine function into a function that returns a Deferred.
The coroutine function will be called at the time when the wrapper is called. Wrapper args will be passed to it.
This is useful for callback chains, as callback functions are called with the previous callback result.
The following function helps with a reverse wrapping:
#### scrapy.utils.defer.ensure_awaitable(o: Awaitable[\_T], \_warn: str | None = None) → Awaitable[\_T]
#### scrapy.utils.defer.ensure_awaitable(o: \_T, \_warn: str | None = None) → Awaitable[\_T]
Convert any value to an awaitable object.
For a `Deferred` object, use
`maybe_deferred_to_future()` to wrap it into a suitable object. For an
awaitable object of a different type, return it as is. For any other
value, return a coroutine that completes with that value.
> [!WARNING]
> Added in version 2.14.
### Enforcing asyncio as a requirement
If you are writing a component that requires asyncio
to work, use `scrapy.utils.asyncio.is_asyncio_available()` to
enforce it as a requirement. For
example:
```python
from scrapy.utils.asyncio import is_asyncio_available
class MyComponent:
def __init__(self):
if not is_asyncio_available():
raise ValueError(
f"{MyComponent.__qualname__} requires the asyncio support. "
f"Make sure you have configured the asyncio reactor in the "
f"TWISTED_REACTOR setting. See the asyncio documentation "
f"of Scrapy for more information."
)
```
#### scrapy.utils.asyncio.is_asyncio_available() → bool
Check if it’s possible to call asyncio code that relies on the asyncio event loop.
> [!WARNING]
> Added in version 2.14.
This function returns `True` if there is a running asyncio event loop. If
there is no such loop, it returns `True` if the Twisted reactor that is
installed is
`AsyncioSelectorReactor`, returns
`False` if a different reactor is installed, and raises a
`RuntimeError` if no reactor is installed.
Code that doesn’t directly require a Twisted reactor should use this
function while code that requires
`AsyncioSelectorReactor` should use
`is_asyncio_reactor_installed()`.
When this returns `True`, an asyncio loop is installed and used by
Scrapy. It’s possible to call functions that require it, such as
`asyncio.sleep()`, and await on `asyncio.Future` objects in
Scrapy-related code.
When this returns `False`, a non-asyncio Twisted reactor is installed.
It’s not possible to use asyncio features that require an asyncio event
loop or await on `asyncio.Future` objects in Scrapy-related code,
but it’s possible to await on `Deferred`
objects.
> [!NOTE]
> As this function uses `asyncio.get_running_loop()`, it will
> only detect the event loop if called in the same thread and from the
> code that runs inside that loop (this shouldn’t be a problem when
> calling it from code such as spiders and Scrapy components, if Scrapy
> is run using one of the supported ways).
> [!WARNING]
> Changed in version 2.15.0: This function now also returns `True` if there is a running asyncio
> loop, even if no Twisted reactor is installed.
#### scrapy.utils.reactor.is_asyncio_reactor_installed() → bool
Check whether the installed reactor is `AsyncioSelectorReactor`.
Raise a `RuntimeError` if no reactor is installed.
In a future Scrapy version, when Scrapy supports running without a Twisted
reactor, this function won’t be useful for checking if it’s possible to use
asyncio features, so the code that that doesn’t directly require a Twisted
reactor should use `scrapy.utils.asyncio.is_asyncio_available()`
instead of this function.
> [!WARNING]
> Changed in version 2.13: In earlier Scrapy versions this function silently installed the default
> reactor if there was no reactor installed. Now it raises an exception to
> prevent silent problems in this case.
### Using Scrapy without a Twisted reactor
> [!WARNING]
> Added in version 2.15.0.
> [!WARNING]
> This is currently experimental and may not be suitable for production use.
> [!NOTE]
> As the Twisted download handlers cannot be used without a reactor,
> the default download handler in this mode is
> `AiohttpDownloadHandler`.
It’s possible to use Scrapy without installing a Twisted reactor at all, by
setting the `TWISTED_REACTOR_ENABLED` setting to `False`. In this
mode Scrapy will use the asyncio event loop directly, and most of the Scrapy
functionality will work in the same way.
Doing this provides several benefits in certain use cases:
* A Twisted reactor, once stopped, cannot be started again. This prevents, for
example, using several instances of
`AsyncCrawlerProcess` in the same process when they
use a reactor, but with `TWISTED_REACTOR_ENABLED=False` it becomes
possible.
* There may be limitations imposed by
`AsyncioSelectorReactor` and related
Twisted code, such as the requirement of using
`SelectorEventLoop` on Windows (see asyncio-windows),
that do not apply if the reactor is not used.
* `AsyncioSelectorReactor` manages the
underlying event loop, and while `AsyncCrawlerRunner`
can use a pre-existing reactor which, in turn, can use a pre-existing event
loop, it’s easier to use `AsyncCrawlerRunner` with a
pre-existing loop directly.
* Omitting the reactor machinery may improve performance and reliability.
#### Limitations
As some Scrapy features and components require a reactor, they don’t work and
are disabled without it. Replacements that don’t require a reactor may be added
in future Scrapy versions. The following features are not available:
* The default HTTP(S) download handler,
`HTTP11DownloadHandler` (this
is likely the biggest difference; Scrapy provides an HTTP(S) download handler
that doesn’t require a reactor and will be used instead of it:
`AiohttpDownloadHandler`)
* `FTPDownloadHandler`
* `H2DownloadHandler`
* topics-telnetconsole
* `CrawlerRunner` and
`CrawlerProcess`
(`AsyncCrawlerProcess` and
`AsyncCrawlerRunner` are available)
* Twisted-specific DNS resolvers (the `TWISTED_DNS_RESOLVER` setting)
* User and 3rd-party code that requires a reactor (see below for examples)
Note that importing Twisted modules and, among other things, creating and using
`Deferred` objects doesn’t require a reactor, so
code that uses `Deferred`,
`Failure` and some other Twisted APIs will not
necessarily stop working.
#### Other differences
When `TWISTED_REACTOR_ENABLED` is set to `False`, Scrapy will change
the defaults of some other settings:
* `TELNETCONSOLE_ENABLED` is set to `False`.
* The `"http"` and `"https"` keys in `DOWNLOAD_HANDLERS_BASE` are
set to `"scrapy.core.downloader.handlers._aiohttp.AiohttpDownloadHandler"`.
* The `"ftp"` key in `DOWNLOAD_HANDLERS_BASE` is set to `None`.
Thus, `AiohttpDownloadHandler`
is used by default for making HTTP(S) requests. Please refer to its
documentation for its differences and limitations compared to
`HTTP11DownloadHandler`.
Additionally, `AsyncCrawlerProcess` will install a
meta path finder that prevents `twisted.internet.reactor` from
being imported. It will be uninstalled when `AsyncCrawlerProcess.start()` exits.
#### Adding support to existing code
Code that doesn’t directly use Twisted APIs or APIs that depend on Twisted ones
doesn’t need special support for running without a reactor.
Here are some examples of APIs and patterns that need a replacement:
* Using `reactor.callLater()` for sleeping or delayed calls.
You can use `asyncio.loop.call_later()` instead.
* Using `twisted.internet.threads.deferToThread()`,
`reactor.callFromThread()` and related APIs to
execute code in other threads. You can use `asyncio.to_thread()`,
`asyncio.loop.call_soon_threadsafe()` and related APIs instead.
* Using `twisted.internet.task.LoopingCall` for scheduling repeated
tasks. As there is no direct replacement in the standard library, you may
need to write your own one using `asyncio.sleep()` in a task.
* Using Twisted network client and server APIs (`reactor.connectTCP()`,
`reactor.listenTCP()`,
`twisted.web.client`, `twisted.mail.smtp` etc.). You can use other
built-in or 3rd-party libraries for this.
* Using `CrawlerProcess` or
`CrawlerRunner`. You should use
`AsyncCrawlerProcess` or
`AsyncCrawlerRunner` respectively instead.
* Checking whether `asyncio` support is available with
`scrapy.utils.reactor.is_asyncio_reactor_installed()`. You should use
`scrapy.utils.asyncio.is_asyncio_available()` instead.
Scrapy provides unified helpers for some of these examples:
#### *async* scrapy.utils.asyncio.sleep(seconds: float) → None
Sleep for *seconds*.
> [!WARNING]
> Added in version 2.18.0.
This uses either `asyncio.sleep()` or
`deferLater()`, depending on whether asyncio
support is available.
#### scrapy.utils.asyncio.call_later(delay: float, func: Callable[[Unpack[\_Ts]], object], \*args: Unpack[\_Ts]) → CallLaterResult
Schedule a function to be called after a delay.
This uses either `asyncio.loop.call_later()` or
`reactor.callLater()`,
depending on whether asyncio support is available.
> [!WARNING]
> Added in version 2.14.0.
#### scrapy.utils.asyncio.create_looping_call(func: ~collections.abc.Callable[[~_P], ~scrapy.utils.asyncio._T], \*args: ~typing.~_P, \*\*kwargs: ~typing.~_P) → AsyncioLoopingCall | LoopingCall
Create an instance of a looping call class.
This creates an instance of
`AsyncioLoopingCall` or
`LoopingCall`, depending on whether asyncio
support is available.
> [!WARNING]
> Added in version 2.14.0.
#### *class* scrapy.utils.asyncio.AsyncioLoopingCall(func: ~collections.abc.Callable[[~_P], ~scrapy.utils.asyncio._T], \*args: ~typing.~_P, \*\*kwargs: ~typing.~_P)
A simple implementation of a periodic call using asyncio, keeping
some API and behavior compatibility with
`LoopingCall`.
The function is called every *interval* seconds, independent of the finish
time of the previous call. If the function is still running when it’s time
to call it again, calls are skipped until the function finishes.
The function must not return a coroutine or a `Deferred`.
> [!WARNING]
> Added in version 2.14.0.
#### *async* scrapy.utils.asyncio.run_in_thread(func: ~collections.abc.Callable[[~_P], ~scrapy.utils.asyncio._T], \*args: ~typing.~_P, \*\*kwargs: ~typing.~_P) → \_T
Call a function in a thread and return its result as a coroutine.
This uses either `asyncio.to_thread()` or
`twisted.internet.threads.deferToThread()`, depending on whether
asyncio support is available.
> [!WARNING]
> Added in version 2.15.0.
If your code needs to know whether the reactor is available, you can either
check for the value of the `TWISTED_REACTOR_ENABLED` setting (you need
access to the `Crawler` instance to do this) or use the
following function:
#### scrapy.utils.reactorless.is_reactorless() → bool
Check if we are running in the reactorless mode, i.e. with
`TWISTED_REACTOR_ENABLED` set to `False`.
As this checks the runtime state and not the setting itself, it can be
wrong when executed very early, before the reactor and/or the asyncio event
loop are initialized.
> [!NOTE]
> As this function uses
> `scrapy.utils.asyncio.is_asyncio_available()`, it has the same
> limitations for detecting a running asyncio event loop as that one.
> [!WARNING]
> Added in version 2.15.0.
In general, code that doesn’t use the reactor (directly or indirectly) can be
used unmodified both with the asyncio reactor and without a reactor. This
includes code that converts Deferreds to futures and vice versa as described in
asyncio-await-dfd.
#### Troubleshooting
**ImportError: Import of twisted.internet.reactor is forbidden when running
without a Twisted reactor […]:** Scrapy is configured to run without a
reactor, but some code imported `twisted.internet.reactor`, most likely
because that code needs a reactor to be used. You need to stop using this code
or set `TWISTED_REACTOR_ENABLED` back to `True`. It’s also possible
that the reactor isn’t really needed but was installed due to the problem
described in asyncio-preinstalled-reactor, in which case it should be
enough to fix the problematic imports.
**RuntimeError: TWISTED_REACTOR_ENABLED is False but a Twisted reactor is
installed:** Scrapy is configured to run without a reactor, but a reactor is
already installed before the Scrapy code is executed. If you are trying to set
`TWISTED_REACTOR_ENABLED` via per-spider settings, it’s currently unsupported.
**RuntimeError: We expected a Twisted reactor to be installed but it isn’t:**
Scrapy is configured to run with a reactor and not to install one, but a
reactor wasn’t installed before the Scrapy code is executed. If you are trying
to set `TWISTED_REACTOR_ENABLED` via per-spider settings, it’s currently unsupported.
**RuntimeError: <class> doesn’t support TWISTED_REACTOR_ENABLED=False:** The
listed class cannot be used with `TWISTED_REACTOR_ENABLED` set to
`False`. There may be a replacement in the documentation above or the documentation of the affected class.
### Windows-specific notes
The Windows implementation of `asyncio` can use two event loop
implementations, `ProactorEventLoop` (default) and
`SelectorEventLoop`. However, only
`SelectorEventLoop` works with Twisted.
Scrapy changes the event loop class to `SelectorEventLoop`
automatically when installing the asyncio reactor.
> [!NOTE]
> Other libraries you use may require
> `ProactorEventLoop`, e.g. because it supports
> subprocesses (this is the case with [playwright](https://github.com/microsoft/playwright-python)), so you cannot use
> them together with Scrapy on Windows (but you should be able to use
> them on WSL or native Linux).
> [!NOTE]
> This problem doesn’t apply when not using the reactor, see
> asyncio-without-reactor.
### Using custom asyncio loops
You can also use custom asyncio event loops with the asyncio reactor. Set the
`ASYNCIO_EVENT_LOOP` setting to the import path of the desired event
loop class to use it instead of the default asyncio event loop.
### Switching to a non-asyncio reactor
If for some reason your code doesn’t work with the asyncio reactor, you can use
a different reactor by setting the `TWISTED_REACTOR` setting to its
import path (e.g. `'twisted.internet.epollreactor.EPollReactor'`) or to
`None`, which will use the default reactor for your platform. If you are
using `AsyncCrawlerRunner` or
`AsyncCrawlerProcess` you also need to switch to their
Deferred-based counterparts: `CrawlerRunner` or
`CrawlerProcess` respectively.
Source: /topics/architecture.md
## Architecture overview
This document describes the architecture of Scrapy and how its components
interact.
### Overview
The following diagram shows an overview of the Scrapy architecture with its
components and an outline of the data flow that takes place inside the system
(shown by the red arrows). A brief description of the components is included
below with links for more detailed information about them. The data flow is
also described below.
### Data flow
The data flow in Scrapy is controlled by the execution engine, and goes like
this:
1. The Engine gets the initial Requests to crawl from the
Spider.
2. The Engine schedules the Requests in the
Scheduler and asks for the
next Requests to crawl.
3. The Scheduler returns the next Requests
to the Engine.
4. The Engine sends the Requests to the
Downloader, passing through the
Downloader Middlewares (see
`process_request()`).
5. Once the page finishes downloading the
Downloader generates a Response (with
that page) and sends it to the Engine, passing through the
Downloader Middlewares (see
`process_response()`).
6. The Engine receives the Response from the
Downloader and sends it to the
Spider for processing, passing
through the Spider Middleware (see
`process_spider_input()`).
7. The Spider processes the Response and returns
scraped items and new Requests (to follow) to the
Engine, passing through the
Spider Middleware (see
`process_spider_output()`).
8. The Engine sends processed items to
Item Pipelines, then sends processed Requests to
the Scheduler and asks for possible next Requests
to crawl.
9. The process repeats (from step 3) until there are no more requests from the
Scheduler.
### Components
#### Scrapy Engine
The engine is responsible for controlling the data flow between all components
of the system, and triggering events when certain actions occur. See the
Data Flow section above for more details.
#### Scheduler
The scheduler receives requests from the engine and
enqueues them for feeding them later (also to the engine) when the engine
requests them.
#### Downloader
The Downloader is responsible for fetching web pages and feeding them to the
engine which, in turn, feeds them to the spiders.
#### Spiders
Spiders are custom classes written by Scrapy users to parse responses and
extract items from them or additional requests to
follow. For more information see topics-spiders.
#### Item Pipeline
The Item Pipeline is responsible for processing the items once they have been
extracted (or scraped) by the spiders. Typical tasks include cleansing,
validation and persistence (like storing the item in a database). For more
information see topics-item-pipeline.
#### Downloader middlewares
Downloader middlewares are specific hooks that sit between the Engine and the
Downloader and process requests when they pass from the Engine to the
Downloader, and responses that pass from Downloader to the Engine. See
concepts for when to use one, and topics-downloader-middleware
for more information.
#### Spider middlewares
Spider middlewares are specific hooks that sit between the Engine and the
Spiders and are able to process spider input (responses) and output (items and
requests). See concepts for when to use one, and
topics-spider-middleware for more information.
#### Extensions
Extensions do not have a specific role in the data flow like the other
components above. See concepts for when to use one, and
topics-extensions for more information.
### Event-driven networking
Scrapy is written with [Twisted](https://twisted.org/), a popular event-driven networking framework
for Python. Thus, it’s implemented using a non-blocking (aka asynchronous) code
for concurrency.
For more information about asynchronous programming and Twisted see these
links:
* twisted:core/howto/defer-intro
* [Twisted Introduction - Krondo](https://krondo.com/an-introduction-to-asynchronous-programming-and-twisted/)
Source: /topics/addons.md
## Add-ons
Scrapy’s add-on system is a framework which unifies managing and configuring
components that extend Scrapy’s core functionality, such as middlewares,
extensions, or pipelines. It provides users with a plug-and-play experience in
Scrapy extension management, and grants extensive configuration control to
developers.
### Activating and configuring add-ons
During `Crawler` initialization, the list of enabled
add-ons is read from your `ADDONS` setting.
The `ADDONS` setting is a dict in which every key is an add-on class or its
import path and the value is its priority.
This is an example where two add-ons are enabled in a project’s
`settings.py`:
```python
ADDONS = {
"path.to.someaddon": 0,
SomeAddonClass: 1,
}
```
### Writing your own add-ons
Add-ons are components that include one or both of
the following methods:
#### update_settings(settings)
This method is called during the initialization of the
`Crawler`. Here, you should perform dependency checks
(e.g. for external Python libraries) and update the
`Settings` object as wished, e.g. enable components
for this add-on or set required configuration of other extensions.
* **Parameters:**
**settings** (`Settings`) – The settings object storing Scrapy/component configuration
#### *classmethod* update_pre_crawler_settings(cls, settings)
Use this class method instead of the `update_settings()` method to
update pre-crawler settings whose value is
used before the `Crawler` object is created.
* **Parameters:**
**settings** (`BaseSettings`) – The settings object storing Scrapy/component configuration
The settings set by the add-on should use the `addon` priority (see
populating-settings and `scrapy.settings.BaseSettings.set()`):
```python
class MyAddon:
def update_settings(self, settings):
settings.set("DNSCACHE_ENABLED", True, "addon")
```
This allows users to override these settings in the project or spider
configuration.
When editing the value of a setting instead of overriding it entirely, it is
usually best to leave its priority unchanged. For example, when editing a
component priority dictionary.
If the `update_settings` method raises
`scrapy.exceptions.NotConfigured`, the add-on will be skipped. This makes
it easy to enable an add-on only when some conditions are met.
#### Fallbacks
Some components provided by add-ons need to fall back to “default”
implementations, e.g. a custom download handler needs to send the request that
it doesn’t handle via the default download handler, or a stats collector that
includes some additional processing but otherwise uses the default stats
collector. And it’s possible that a project needs to use several custom
components of the same type, e.g. two custom download handlers that support
different kinds of custom requests and still need to use the default download
handler for other requests. To make such use cases easier to configure, we
recommend that such custom components should be written in the following way:
1. The custom component (e.g. `MyDownloadHandler`) shouldn’t inherit from the
default Scrapy one (e.g.
`scrapy.core.downloader.handlers.http11.HTTP11DownloadHandler`), but instead
be able to load the class of the fallback component from a special setting
(e.g. `MY_FALLBACK_DOWNLOAD_HANDLER`), create an instance of it and use
it.
2. The add-ons that include these components should read the current value of
the default setting (e.g. `DOWNLOAD_HANDLERS`) in their
`update_settings()` methods, save that value into the fallback setting
(`MY_FALLBACK_DOWNLOAD_HANDLER` mentioned earlier) and set the default
setting to the component provided by the add-on (e.g.
`MyDownloadHandler`). If the fallback setting is already set by the user,
it should not be changed.
3. This way, if there are several add-ons that want to modify the same setting,
all of them will fall back to the component from the previous one and then to
the Scrapy default. The order of that depends on the priority order in the
`ADDONS` setting.
### Add-on examples
Set some basic configuration:
```python
from myproject.pipelines import MyPipeline
class MyAddon:
def update_settings(self, settings):
settings.set("DNSCACHE_ENABLED", True, "addon")
settings.remove_from_list("METAREFRESH_IGNORE_TAGS", "noscript")
settings.setdefault_in_component_priority_dict(
"ITEM_PIPELINES", MyPipeline, 200
)
```
> [!TIP]
> When editing a component priority dictionary setting, like `ITEM_PIPELINES`,
> consider using setting methods like
> `replace_in_component_priority_dict()`,
> `set_in_component_priority_dict()`
> and
> `setdefault_in_component_priority_dict()`
> to avoid mistakes.
Check dependencies:
```python
class MyAddon:
def update_settings(self, settings):
try:
import boto
except ImportError:
raise NotConfigured("MyAddon requires the boto library")
...
```
Access the crawler instance:
```python
class MyAddon:
def __init__(self, crawler) -> None:
super().__init__()
self.crawler = crawler
@classmethod
def from_crawler(cls, crawler):
return cls(crawler)
def update_settings(self, settings): ...
```
Use a fallback component:
```python
from scrapy.utils.misc import build_from_crawler, load_object
FALLBACK_SETTING = "MY_FALLBACK_DOWNLOAD_HANDLER"
class MyHandler:
lazy = False
def __init__(self, crawler):
dhcls = load_object(crawler.settings.get(FALLBACK_SETTING))
self._fallback_handler = build_from_crawler(dhcls, crawler)
async def download_request(self, request):
if request.meta.get("my_params"):
# handle the request
...
else:
return await self._fallback_handler.download_request(request)
async def close(self):
pass
class MyAddon:
def update_settings(self, settings):
if not settings.get(FALLBACK_SETTING):
settings.set(
FALLBACK_SETTING,
settings.getwithbase("DOWNLOAD_HANDLERS")["https"],
"addon",
)
settings["DOWNLOAD_HANDLERS"]["https"] = MyHandler
```
Source: /topics/downloader-middleware.md
## Downloader Middleware
The downloader middleware is a framework of hooks into Scrapy’s
request/response processing. It’s a light, low-level system for globally
altering Scrapy’s requests and responses at the HTTP level. See
concepts for a rundown of other alternatives.
### Activating a downloader middleware
To activate a downloader middleware component, add it to the
`DOWNLOADER_MIDDLEWARES` setting, which is a dict whose keys are the
middleware class paths and their values are the middleware orders.
Here’s an example:
```python
DOWNLOADER_MIDDLEWARES = {
"myproject.middlewares.CustomDownloaderMiddleware": 543,
}
```
The `DOWNLOADER_MIDDLEWARES` setting is merged with the
`DOWNLOADER_MIDDLEWARES_BASE` setting defined in Scrapy (and not meant
to be overridden) and then sorted by order to get the final sorted list of
enabled middlewares: the first middleware is the one closer to the engine and
the last is the one closer to the downloader. In other words,
the `process_request()`
method of each middleware will be invoked in increasing
middleware order (100, 200, 300, …) and the `process_response()` method
of each middleware will be invoked in decreasing order.
To decide which order to assign to your middleware see the
`DOWNLOADER_MIDDLEWARES_BASE` setting and pick a value according to
where you want to insert the middleware. The order does matter because each
middleware performs a different action and your middleware could depend on some
previous (or subsequent) middleware being applied.
If you want to disable a built-in middleware (the ones defined in
`DOWNLOADER_MIDDLEWARES_BASE` and enabled by default) you must define it
in your project’s `DOWNLOADER_MIDDLEWARES` setting and assign `None`
as its value. For example, if you want to disable the user-agent middleware:
```python
DOWNLOADER_MIDDLEWARES = {
"myproject.middlewares.CustomDownloaderMiddleware": 543,
"scrapy.downloadermiddlewares.useragent.UserAgentMiddleware": None,
}
```
Finally, keep in mind that some middlewares may need to be enabled through a
particular setting. See each middleware documentation for more info.
### Writing your own downloader middleware
Each downloader middleware is a component that
defines one or more of these methods:
#### *class* scrapy.downloadermiddlewares.DownloaderMiddleware
> [!NOTE]
> Any of the downloader middleware methods may be defined as a
> coroutine function (`async def`).
##### process_request(request)
This method is called for each request that goes through the download
middleware.
`process_request()` should either: return `None`, return a
`Response` object, return a `Request`
object, or raise `IgnoreRequest`.
If it returns `None`, Scrapy will continue processing this request, executing all
other middlewares until, finally, the appropriate downloader handler is called
the request performed (and its response downloaded).
If it returns a `Response` object, Scrapy won’t bother
calling *any* other `process_request()` or `process_exception()` methods,
or the appropriate download function; it’ll return that response. The `process_response()`
methods of installed middleware is always called on every response.
If it returns a `Request` object, Scrapy will stop calling
`process_request()` methods and reschedule the returned request. Once the newly returned
request is performed, the appropriate middleware chain will be called on
the downloaded response.
If it raises an `IgnoreRequest` exception, the
`process_exception()` methods of installed downloader middleware will be called.
If none of them handle the exception, the errback function of the request
(`Request.errback`) is called. If no code handles the raised exception, it is
ignored and not logged (unlike other exceptions).
* **Parameters:**
**request** (`Request` object) – the request being processed
##### process_response(request, response)
`process_response()` should either: return a `Response`
object, return a `Request` object or
raise a `IgnoreRequest` exception.
If it returns a `Response` (it could be the same given
response, or a brand-new one), that response will continue to be processed
with the `process_response()` of the next middleware in the chain.
If it returns a `Request` object, the middleware chain is
halted and the returned request is rescheduled to be downloaded in the future.
This is the same behavior as if a request is returned from `process_request()`.
If it raises an `IgnoreRequest` exception, the errback
function of the request (`Request.errback`) is called. If no code handles the raised
exception, it is ignored and not logged (unlike other exceptions).
* **Parameters:**
* **request** (is a `Request` object) – the request that originated the response
* **response** (`Response` object) – the response being processed
##### process_exception(request, exception)
Scrapy calls `process_exception()` when a download handler or a `process_request()` (from a
downloader middleware) raises an exception (including an
`IgnoreRequest` exception).
`process_exception()` should return: either `None`,
a `Response` object, or a `Request` object.
If it returns `None`, Scrapy will continue processing this exception,
executing any other `process_exception()` methods of installed middleware,
until no middleware is left and the default exception handling kicks in.
If it returns a `Response` object, the `process_response()`
method chain of installed middleware is started, and Scrapy won’t bother calling
any other `process_exception()` methods of middleware.
If it returns a `Request` object, the returned request is
rescheduled to be downloaded in the future. This stops the execution of
`process_exception()` methods of the middleware the same as returning a
response would.
* **Parameters:**
* **request** (is a `Request` object) – the request that generated the exception
* **exception** (an `Exception` object) – the raised exception
### Downloading a request from a downloader middleware
A downloader middleware can download a request of its own while it processes
another one, e.g. to fetch something that the request it is processing needs.
The built-in robots.txt middleware does that: it
holds each request while it downloads the `robots.txt` file of its website.
Use `crawler.engine.download_async()` for that:
```python
from scrapy import Request
from scrapy.http.request import NO_CALLBACK
class TokenMiddleware:
def __init__(self, crawler):
self.crawler = crawler
self.token = None
@classmethod
def from_crawler(cls, crawler):
return cls(crawler)
async def process_request(self, request):
if request.meta.get("dont_obey_robotstxt"):
return
if self.token is None:
response = await self.crawler.engine.download_async(
Request(
"https://example.com/token",
callback=NO_CALLBACK,
meta={"dont_obey_robotstxt": True},
)
)
self.token = response.text
request.headers["Authorization"] = self.token
```
Requests that you download this way go through the downloader middleware chain
as well, including your own middleware and the robots.txt middleware, which holds a request until the `robots.txt` file of
its website arrives. Be careful not to introduce deadlocks: a request that you
download must not end up waiting for the request that is waiting for it. Hence
`dont_obey_robotstxt` above, which makes both middlewares let the token
request through.
While the first token response is in transit, `process_request` runs for other
requests as well, and the middleware above downloads a token for each of them.
Cache the task that downloads the token, and not only its result, to download
the token only once.
### Built-in downloader middleware reference
This page describes all downloader middleware components that come with
Scrapy. For information on how to use them and how to write your own downloader
middleware, see the downloader middleware usage guide.
For a list of the components enabled by default (and their orders) see the
`DOWNLOADER_MIDDLEWARES_BASE` setting.
#### CookiesMiddleware
See cookies.
#### DefaultHeadersMiddleware
#### *class* scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware
This middleware sets all default requests headers specified in the
`DEFAULT_REQUEST_HEADERS` setting.
#### DownloadTimeoutMiddleware
#### *class* scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware
This middleware sets the download timeout for requests specified in the
`DOWNLOAD_TIMEOUT` setting.
> [!NOTE]
> You can also set download timeout per-request using the
> `download_timeout` `Request.meta` key; this is supported
> even when DownloadTimeoutMiddleware is disabled.
#### HttpAuthMiddleware
#### *class* scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware
This middleware authenticates requests using [Basic access authentication](https://en.wikipedia.org/wiki/Basic_access_authentication)
(aka. HTTP auth).
Use the `HTTPAUTH_USER`, `HTTPAUTH_PASS`, and
`HTTPAUTH_DOMAIN` settings to configure it. You can also override
the credentials per request via `meta` keys
`http_user`, `http_pass`, and `http_auth_domain`.
Example using settings (e.g. in `custom_settings`):
```python
from scrapy.spiders import CrawlSpider
class SomeIntranetSiteSpider(CrawlSpider):
name = "intranet.example.com"
custom_settings = {
"HTTPAUTH_USER": "someuser",
"HTTPAUTH_PASS": "somepass",
"HTTPAUTH_DOMAIN": "intranet.example.com",
}
# .. rest of the spider code omitted ...
```
Example using per-request meta:
```python
async def start(self):
yield Request(
"https://intranet.example.com/protected/",
meta={
"http_user": "someuser",
"http_pass": "somepass",
"http_auth_domain": "intranet.example.com",
},
)
```
##### HTTPAUTH_USER
> [!WARNING]
> Added in version 2.17.0.
Default: `""`
The username to use for HTTP basic authentication, applied to all requests
whose URL matches `HTTPAUTH_DOMAIN`.
##### HTTPAUTH_PASS
> [!WARNING]
> Added in version 2.17.0.
Default: `""`
The password to use for HTTP basic authentication.
##### HTTPAUTH_DOMAIN
> [!WARNING]
> Added in version 2.17.0.
Default: `None`
The domain (and its subdomains) to which HTTP basic authentication credentials
are sent. Set to `None` to send credentials with all requests, but be aware
that this risks leaking credentials to unrelated domains.
This setting must be explicitly configured whenever `HTTPAUTH_USER`
or `HTTPAUTH_PASS` is set.
> [!NOTE]
> ### See also
>
> security-credential-leakage
#### HttpCacheMiddleware
#### *class* scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware
This middleware provides low-level cache to all HTTP requests and responses.
It has to be combined with a cache storage backend as well as a cache policy.
Scrapy ships with the following HTTP cache storage backends:
> * httpcache-storage-fs
> * httpcache-storage-dbm
You can change the HTTP cache storage backend with the `HTTPCACHE_STORAGE`
setting. Or you can also implement your own storage backend.
Scrapy ships with two HTTP cache policies:
> * httpcache-policy-rfc2616
> * httpcache-policy-dummy
You can change the HTTP cache policy with the `HTTPCACHE_POLICY`
setting. Or you can also implement your own policy.
You can also avoid caching a response on every policy using `dont_cache` meta key equals `True`.
> [!WARNING]
> Added in version 2.19.0.
When a response comes from the cache, its `meta`
exposes the `cache_timestamp` key: the Unix timestamp of when the
response was stored, useful e.g. to interpret dates relative to the time
the page was downloaded rather than to the current time.
##### Dummy policy (default)
#### *class* scrapy.extensions.httpcache.DummyPolicy
This policy has no awareness of any HTTP Cache-Control directives.
Every request and its corresponding response are cached. When the same
request is seen again, the response is returned without transferring
anything from the Internet.
The Dummy policy is useful for testing spiders faster (without having
to wait for downloads every time) and for trying your spider offline,
when an Internet connection is not available. The goal is to be able to
“replay” a spider run *exactly as it ran before*.
##### RFC2616 policy
#### *class* scrapy.extensions.httpcache.RFC2616Policy
This policy provides a RFC2616 compliant HTTP cache, i.e. with HTTP
Cache-Control awareness, aimed at production and used in continuous
runs to avoid downloading unmodified data (to save bandwidth and speed up
crawls).
What is implemented:
* Do not attempt to store responses/requests with `no-store` cache-control directive set
* Do not serve responses from cache if `no-cache` cache-control directive is set even for fresh responses
* Compute freshness lifetime from `max-age` cache-control directive
* Compute freshness lifetime from `Expires` response header
* Compute freshness lifetime from `Last-Modified` response header (heuristic used by Firefox)
* Compute current age from `Age` response header
* Compute current age from `Date` header
* Revalidate stale responses based on `Last-Modified` response header
* Revalidate stale responses based on `ETag` response header
* Set `Date` header for any received response missing it
* Support `max-stale` cache-control directive in requests
This allows spiders to be configured with the full RFC2616 cache policy,
but avoid revalidation on a request-by-request basis, while remaining
conformant with the HTTP spec.
Example:
Add `Cache-Control: max-stale=600` to Request headers to accept responses that
have exceeded their expiration time by no more than 600 seconds.
See also: RFC2616, 14.9.3
What is missing:
* `Pragma: no-cache` support [https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.9.1](https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.9.1)
* `Vary` header support [https://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.6](https://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.6)
* Invalidation after updates or deletes [https://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.10](https://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.10)
* … probably others ..
##### Filesystem storage backend (default)
#### *class* scrapy.extensions.httpcache.FilesystemCacheStorage
File system storage backend is available for the HTTP cache middleware.
Each request/response pair is stored in a different directory containing
the following files:
* `request_body` - the plain request body
* `request_headers` - the request headers (in raw HTTP format)
* `response_body` - the plain response body
* `response_headers` - the response headers (in raw HTTP format)
* `response_data` - the remaining data of the response, as returned by
`Response.to_dict()`, pickled
* `meta` - some metadata of this cache resource in Python `repr()`
format (grep-friendly format)
* `pickled_meta` - the same metadata in `meta` but pickled for more
efficient deserialization
The directory name is made from the request fingerprint (see
`scrapy.utils.request.fingerprint`), and one level of subdirectories is
used to avoid creating too many files into the same directory (which is
inefficient in many file systems). An example directory could be:
```
/path/to/cache/dir/example.com/72/72811f648e718090f041317756c03adb0ada46c7
```
##### DBM storage backend
#### *class* scrapy.extensions.httpcache.DbmCacheStorage
A [DBM](https://en.wikipedia.org/wiki/Dbm) storage backend is also available for the HTTP cache middleware.
By default, it uses the `dbm`, but you can change it with the
`HTTPCACHE_DBM_MODULE` setting.
##### Writing your own storage backend
You can implement a cache storage backend by creating a Python class that
defines the methods described below.
To store a response, use `Response.to_dict()`, and to read it back, use
`response_from_dict()`. That way responses of any
class, including those of third-party plugins, are cached and restored intact.
#### *class* scrapy.extensions.httpcache.CacheStorage
##### open_spider(spider)
This method gets called after a spider has been opened for crawling. It handles
the `spider_opened` signal.
* **Parameters:**
**spider** (`Spider` object) – the spider which has been opened
##### close_spider(spider)
This method gets called after a spider has been closed. It handles
the `spider_closed` signal.
* **Parameters:**
**spider** (`Spider` object) – the spider which has been closed
##### retrieve_response(spider, request)
Return response if present in cache, or `None` otherwise.
If this method raises an exception, e.g. because the cache entry is
corrupted, the middleware logs a warning and handles the request as a
cache miss.
* **Parameters:**
* **spider** (`Spider` object) – the spider which generated the request
* **request** (`Request` object) – the request to find cached response for
On a cache hit, set `request.meta["cache_timestamp"]` (see
`cache_timestamp`) to the Unix timestamp of when the response
was stored.
##### store_response(spider, request, response)
Store the given response in the cache.
* **Parameters:**
* **spider** (`Spider` object) – the spider for which the response is intended
* **request** (`Request` object) – the corresponding request the spider generated
* **response** (`Response` object) – the response to store in the cache
In order to use your storage backend, set:
* `HTTPCACHE_STORAGE` to the Python import path of your custom storage class.
##### HTTPCache middleware settings
`HttpCacheMiddleware` can be
configured through the following settings:
###### HTTPCACHE_ENABLED
Default: `False`
Whether the HTTP cache will be enabled.
###### HTTPCACHE_EXPIRATION_SECS
Default: `0`
Expiration time for cached requests, in seconds.
Cached requests older than this time will be re-downloaded. If zero, cached
requests will never expire.
###### HTTPCACHE_DIR
Default: `'httpcache'`
The directory to use for storing the (low-level) HTTP cache. If empty, the HTTP
cache will be disabled. If a relative path is given, is taken relative to the
project data dir. For more info see: topics-project-structure.
###### HTTPCACHE_IGNORE_HTTP_CODES
Default: `[]`
Don’t cache response with these HTTP codes.
###### HTTPCACHE_IGNORE_MISSING
Default: `False`
If enabled, requests not found in the cache will be ignored instead of downloaded.
###### HTTPCACHE_IGNORE_SCHEMES
Default: `['file']`
Don’t cache responses with these URI schemes.
###### HTTPCACHE_STORAGE
Default: `'scrapy.extensions.httpcache.FilesystemCacheStorage'`
The class which implements the cache storage backend.
###### HTTPCACHE_DBM_MODULE
Default: `'dbm'`
The database module to use in the DBM storage backend. This setting is specific to the DBM backend.
###### HTTPCACHE_POLICY
Default: `'scrapy.extensions.httpcache.DummyPolicy'`
The class which implements the cache policy.
###### HTTPCACHE_GZIP
Default: `False`
If enabled, will compress all cached data with gzip.
This setting is specific to the Filesystem backend.
###### HTTPCACHE_ALWAYS_STORE
Default: `False`
If enabled, will cache pages unconditionally.
A spider may wish to have all responses available in the cache, for
future use with `Cache-Control: max-stale`, for instance. The
DummyPolicy caches all responses but never revalidates them, and
sometimes a more nuanced policy is desirable.
This setting still respects `Cache-Control: no-store` directives in responses.
If you don’t want that, filter `no-store` out of the Cache-Control headers in
responses you feed to the cache middleware.
###### HTTPCACHE_IGNORE_RESPONSE_CACHE_CONTROLS
Default: `[]`
List of Cache-Control directives in responses to be ignored.
Sites often set “no-store”, “no-cache”, “must-revalidate”, etc., but get
upset at the traffic a spider can generate if it actually respects those
directives. This allows to selectively ignore Cache-Control directives
that are known to be unimportant for the sites being crawled.
We assume that the spider will not issue Cache-Control directives
in requests unless it actually needs them, so directives in requests are
not filtered.
#### HttpCompressionMiddleware
#### *class* scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware
This middleware allows compressed (gzip, deflate, [brotli](https://www.ietf.org/rfc/rfc7932.txt), [zstd](https://www.ietf.org/rfc/rfc8478.txt))
traffic to be sent/received from web sites.
##### HttpCompressionMiddleware Settings
###### COMPRESSION_ENABLED
Default: `True`
Whether the Compression middleware will be enabled.
#### HttpProxyMiddleware
#### *class* scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware
This middleware sets the HTTP proxy to use for requests, by setting the
`proxy` meta value for `Request` objects.
Like the Python standard library module `urllib.request`, it obeys
the following environment variables:
* `http_proxy`
* `https_proxy`
* `no_proxy`
You can also set the meta key `proxy` per-request, to a value like
`http://some_proxy_server:port` or `http://username:password@some_proxy_server:port`.
Keep in mind this value will take precedence over `http_proxy`/`https_proxy`
environment variables, and it will also ignore `no_proxy` environment variable.
> [!NOTE]
> Handling of this meta key needs to be implemented inside the download
> handler, so it’s not guaranteed to be supported
> by all 3rd-party handlers. It’s currently unsupported by
> `H2DownloadHandler`.
> [!NOTE]
> Usually a proxy URL uses the `http://` scheme. More rarely, it uses the
> `https://` one. While both kinds of proxy URLs can be used with both HTTP
> and HTTPS destination URLs, the specifics of the network exchange are
> different for all 4 cases and it’s possible that HTTPS proxies are fully or
> partially unsupported by a given download handler. Currently,
> `HTTP11DownloadHandler`
> supports HTTPS proxies only for HTTP destinations.
> [!NOTE]
> If the download handler supports it, you can use a SOCKS proxy URL (e.g.
> `socks5://username:password@some_proxy_server:port`).
> `HttpxDownloadHandler`
> supports SOCKS proxies while other built-in handlers don’t.
##### HttpProxyMiddleware settings
###### HTTPPROXY_ENABLED
Default: `True`
Whether or not to enable the `HttpProxyMiddleware`.
###### HTTPPROXY_AUTH_ENCODING
Default: `"latin-1"`
The default encoding for proxy authentication on `HttpProxyMiddleware`.
#### OffsiteMiddleware
#### *class* scrapy.downloadermiddlewares.offsite.OffsiteMiddleware(stats: StatsCollector)
Filter out requests for URLs outside the domains covered by the spider.
> [!WARNING]
> Added in version 2.11.2.
A request is allowed if its host name is in the
`allowed_domains` attribute of the spider, or is a
subdomain of one of those domains. E.g. `www.example.org` also allows
`bob.www.example.org`, but neither `www2.example.org` nor
`example.org`. See `should_follow()` to use a different policy.
If the spider does not define `allowed_domains`, or
the attribute is empty, every request is allowed.
Filtered requests are logged as follows:
```
DEBUG: Filtered offsite request to 'offsite.example': <GET http://offsite.example/some/page.html>
```
Only the first request filtered for a given domain is logged, to keep the
log readable.
##### allow_offsite
Requests with the `allow_offsite` `meta` key set to
`True`, or with `dont_filter` set to `True`, are
allowed regardless of their host name.
##### should_follow(request: Request, spider: Spider) → bool
Return `True` if *request* is on site, `False` if it must be
filtered out.
Override this method to implement a different offsite policy. For
example, to allow the domains in
`allowed_domains` but none of their subdomains:
```python
from scrapy.downloadermiddlewares.offsite import OffsiteMiddleware
from scrapy.utils.httpobj import urlparse_cached
class RootOnlyOffsiteMiddleware(OffsiteMiddleware):
def should_follow(self, request, spider):
return urlparse_cached(request).hostname in spider.allowed_domains
```
#### RedirectMiddleware
#### *class* scrapy.downloadermiddlewares.redirect.RedirectMiddleware
This middleware handles redirection of requests based on response status.
The urls which the request goes through (while being redirected) can be found
in the `redirect_urls` `Request.meta` key.
The reason behind each redirect in `redirect_urls` can be found in the
`redirect_reasons` `Request.meta` key. For
example: `[301, 302, 307, 'meta refresh']`.
The format of a reason depends on the middleware that handled the corresponding
redirect. For example, `RedirectMiddleware` indicates the triggering
response status code as an integer, while `MetaRefreshMiddleware`
always uses the `'meta refresh'` string as reason.
The number of redirects that a request has gone through so far can be found in
the `redirect_times` `Request.meta` key.
The number of redirects that a request may still go through is tracked in the
`redirect_ttl` `Request.meta` key, which is
initialized from `REDIRECT_MAX_TIMES` and decreased on every redirect.
Set it on a request to allow fewer redirects for that request;
`REDIRECT_MAX_TIMES` still applies as an upper bound.
The `RedirectMiddleware` can be configured through the following
settings (see the settings documentation for more info):
* `REDIRECT_ENABLED`
* `REDIRECT_MAX_TIMES`
If `Request.meta` has `dont_redirect`
key set to True, the request will be ignored by this middleware.
If you want to handle some redirect status codes in your spider, you can
specify these in the `handle_httpstatus_list` spider attribute.
For example, if you want the redirect middleware to ignore 301 and 302
responses (and pass them through to your spider) you can do this:
```python
class MySpider(CrawlSpider):
handle_httpstatus_list = [301, 302]
```
The `handle_httpstatus_list` key of `Request.meta` can also be used to specify which response codes to
allow on a per-request basis. You can also set the meta key
`handle_httpstatus_all` to `True` if you want to allow any response code
for a request.
To decide whether to follow a redirect based on its target URL, subclass
`RedirectMiddleware` and override
`process_response()`:
```python
import re
from scrapy import Request
from scrapy.downloadermiddlewares.redirect import RedirectMiddleware
from scrapy.exceptions import IgnoreRequest
class AllowedRedirectMiddleware(RedirectMiddleware):
def process_response(self, request, response, spider):
result = super().process_response(request, response, spider)
if isinstance(result, Request) and not re.search(r"/product/", result.url):
raise IgnoreRequest(f"Redirect target not allowed: {result.url}")
return result
```
Enable it in place of `RedirectMiddleware`, using the same priority:
```python
from scrapy import Spider
class MySpider(Spider):
custom_settings = {
"DOWNLOADER_MIDDLEWARES": {
"scrapy.downloadermiddlewares.redirect.RedirectMiddleware": None,
"myproject.middlewares.AllowedRedirectMiddleware": 600,
}
}
```
##### RedirectMiddleware settings
###### REDIRECT_ENABLED
Default: `True`
Whether the Redirect middleware will be enabled.
###### REDIRECT_MAX_TIMES
Default: `20`
The maximum number of redirections that will be followed for a single request.
If maximum redirections are exceeded, the request is aborted and ignored.
#### MetaRefreshMiddleware
#### *class* scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware
This middleware handles redirection of requests based on meta-refresh html tag.
The `MetaRefreshMiddleware` can be configured through the following
settings (see the settings documentation for more info):
* `METAREFRESH_ENABLED`
* `METAREFRESH_IGNORE_TAGS`
* `METAREFRESH_MAXDELAY`
This middleware obey `REDIRECT_MAX_TIMES` setting, `dont_redirect`,
`redirect_urls` and `redirect_reasons` request meta keys as described
for `RedirectMiddleware`
##### MetaRefreshMiddleware settings
###### METAREFRESH_ENABLED
Default: `True`
Whether the Meta Refresh middleware will be enabled.
###### METAREFRESH_IGNORE_TAGS
Default: `["noscript"]`
Meta tags within these tags are ignored.
> [!WARNING]
> Changed in version 2.11.2: The default value of `METAREFRESH_IGNORE_TAGS` changed from
> `[]` to `["noscript"]`.
###### METAREFRESH_MAXDELAY
Default: `100`
The maximum meta-refresh delay (in seconds) to follow the redirection.
Some sites use meta-refresh for redirecting to a session expired page, so we
restrict automatic redirection to the maximum delay.
#### RetryMiddleware
#### *class* scrapy.downloadermiddlewares.retry.RetryMiddleware
A middleware to retry failed requests that are potentially caused by
temporary problems such as a connection timeout or HTTP 500 error.
If `Request.meta` has `dont_retry` key
set to True, the request will be ignored by this middleware.
The number of times that a request has been retried so far can be found in the
`retry_times` `Request.meta` key. Since it counts
against `RETRY_TIMES`, copying it into a follow-up request lowers the
number of retries allowed for that request.
The `priority_adjust` `Request.meta` key
overrides `RETRY_PRIORITY_ADJUST` for a request.
To retry requests from a spider callback, you can use the
`get_retry_request()` function:
#### scrapy.downloadermiddlewares.retry.get_retry_request(request: Request, \*, spider: scrapy.Spider, reason: str | Exception | type[Exception] = 'unspecified', max_retry_times: int | None = None, priority_adjust: int | None = None, logger: Logger = <Logger scrapy.downloadermiddlewares.retry (WARNING)>, give_up_log_level: int | str | None = None, stats_base_key: str = 'retry') → Request | None
Returns a new `Request` object to retry the specified
request, or `None` if retries of the specified request have been
exhausted.
For example, in a `Spider` callback, you could use it as
follows:
```python
def parse(self, response):
if not response.text:
new_request_or_none = get_retry_request(
response.request,
spider=self,
reason="empty",
)
return new_request_or_none
```
*spider* is the `Spider` instance which is asking for the
retry request. It is used to access the settings
and stats, and to provide extra logging context (see
`logging.debug()`).
*reason* is a string, an `Exception` subclass or an
`Exception` object that indicates the reason why the request needs
to be retried. It is used to name retry stats.
*max_retry_times* is a number that determines the maximum number of times
that *request* can be retried. If not specified or `None`, the number is
read from the `max_retry_times` meta key of the request. If the
`max_retry_times` meta key is not defined or `None`, the number
is read from the `RETRY_TIMES` setting.
*priority_adjust* is a number that determines how the priority of the new
request changes in relation to *request*. If not specified, the number is
read from the `RETRY_PRIORITY_ADJUST` setting.
*logger* is the logging.Logger object to be used when logging messages
*give_up_log_level* is the logging level used for the
message logged when a request exceeds its retries. See
`RETRY_GIVE_UP_LOG_LEVEL` for details.
> [!WARNING]
> Added in version 2.17.0: The *give_up_log_level* parameter.
*stats_base_key* is a string to be used as the base key for the
retry-related job stats
##### RetryMiddleware Settings
###### RETRY_ENABLED
Default: `True`
Whether the Retry middleware will be enabled.
###### RETRY_TIMES
Default: `2`
Maximum number of times to retry, in addition to the first download.
Maximum number of retries can also be specified per-request using
`max_retry_times` attribute of `Request.meta`.
When initialized, the `max_retry_times` meta key takes higher
precedence over the `RETRY_TIMES` setting.
###### RETRY_HTTP_CODES
Default: `[500, 502, 503, 504, 522, 524, 408, 429]`
Which HTTP response codes to retry. Other errors (DNS lookup issues,
connections lost, etc) are always retried.
In some cases you may want to add 400 to `RETRY_HTTP_CODES` because
it is a common code used to indicate server overload. It is not included by
default because HTTP specs say so.
###### RETRY_EXCEPTIONS
Default:
```
[
'scrapy.exceptions.CannotResolveHostError',
'scrapy.exceptions.DownloadConnectionRefusedError',
'scrapy.exceptions.DownloadFailedError',
'scrapy.exceptions.DownloadTimeoutError',
'scrapy.exceptions.ResponseDataLossError',
'twisted.internet.error.ConnectionDone',
'twisted.internet.error.ConnectError',
'twisted.internet.error.ConnectionLost',
OSError,
'scrapy.core.downloader.handlers.http11.TunnelError',
]
```
List of exceptions to retry.
Each list entry may be an exception type or its import path as a string.
An exception will not be caught when the exception type is not in
`RETRY_EXCEPTIONS` or when the maximum number of retries for a request
has been exceeded (see `RETRY_TIMES`). To learn about uncaught
exception propagation, see
`process_exception()`.
###### RETRY_GIVE_UP_LOG_LEVEL
> [!WARNING]
> Added in version 2.17.0.
Default: `"ERROR"`
Logging level used for the message logged when a request
exceeds its retries.
Can be a level name (e.g. `"WARNING"`) or a number (e.g. `logging.WARNING`
or `30`).
See also: `give_up_log_level`, `get_retry_request()`.
###### RETRY_PRIORITY_ADJUST
Default: `-1`
Adjust retry request priority relative to original request:
- a positive priority adjust means higher priority.
- **a negative priority adjust (default) means lower priority.**
See also: `priority_adjust`.
#### RobotsTxtMiddleware
#### *class* scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware
This middleware filters out requests forbidden by the robots.txt exclusion
standard.
To make sure Scrapy respects robots.txt make sure the middleware is enabled
and the `ROBOTSTXT_OBEY` setting is enabled.
The `ROBOTSTXT_USER_AGENT` setting can be used to specify the
user agent string to use for matching in the [robots.txt](https://www.robotstxt.org/) file. If it
is `None`, the User-Agent header you are sending with the request or the
`USER_AGENT` setting (in that order) will be used for determining
the user agent to use in the [robots.txt](https://www.robotstxt.org/) file.
This middleware has to be combined with a [robots.txt](https://www.robotstxt.org/) parser.
Scrapy ships with support for the following [robots.txt](https://www.robotstxt.org/) parsers:
* Protego (default)
* RobotFileParser
* Robotexclusionrulesparser
You can change the [robots.txt](https://www.robotstxt.org/) parser with the `ROBOTSTXT_PARSER`
setting. Or you can also implement support for a new parser.
If `Request.meta` has
`dont_obey_robotstxt` key set to True
the request will be ignored by this middleware even if
`ROBOTSTXT_OBEY` is enabled.
Parsers vary in several aspects:
* Language of implementation
* Supported specification
* Support for wildcard matching
* Usage of [length based rule](https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec#order-of-precedence-for-rules):
in particular for `Allow` and `Disallow` directives, where the most
specific rule based on the length of the path trumps the less specific
(shorter) rule
Performance comparison of different parsers is available at [the following link](https://github.com/scrapy/scrapy/issues/3969).
##### Protego parser
Based on [Protego](https://github.com/scrapy/protego):
* implemented in Python
* is compliant with [Google’s Robots.txt Specification](https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec)
* supports wildcard matching
* uses the length based rule
Scrapy uses this parser by default.
##### RobotFileParser
Based on `RobotFileParser`:
* is Python’s built-in [robots.txt](https://www.robotstxt.org/) parser
* is compliant with [Martijn Koster’s 1996 draft specification](https://www.robotstxt.org/norobots-rfc.txt)
* lacks support for wildcard matching (before Python 3.14.5)
* doesn’t use the length based rule (before Python 3.14.5)
It is faster than Protego and backward-compatible with versions of Scrapy before 1.8.0.
In order to use this parser, set:
* `ROBOTSTXT_PARSER` to `scrapy.robotstxt.PythonRobotParser`
##### Robotexclusionrulesparser
Based on [Robotexclusionrulesparser](https://pypi.org/project/robotexclusionrulesparser/):
* implemented in Python
* is compliant with [Martijn Koster’s 1996 draft specification](https://www.robotstxt.org/norobots-rfc.txt)
* supports wildcard matching
* doesn’t use the length based rule
In order to use this parser:
* Install the robotparser extra.
* Set `ROBOTSTXT_PARSER` setting to
`scrapy.robotstxt.RerpRobotParser`
##### Implementing support for a new parser
You can implement support for a new [robots.txt](https://www.robotstxt.org/) parser by subclassing
the abstract base class `RobotParser` and
implementing the methods described below.
#### *class* scrapy.robotstxt.RobotParser
##### *abstractmethod classmethod* from_crawler(crawler: Crawler, robotstxt_body: bytes) → Self
Parse the content of a [robots.txt](https://www.robotstxt.org/) file as bytes. This must be a class method.
It must return a new instance of the parser backend.
* **Parameters:**
* **crawler** (`Crawler` instance) – crawler which made the request
* **robotstxt_body** (*bytes*) – content of a [robots.txt](https://www.robotstxt.org/) file.
##### *abstractmethod* allowed(url: str | bytes, user_agent: str | bytes) → bool
Return `True` if `user_agent` is allowed to crawl `url`, otherwise return `False`.
* **Parameters:**
* **url** (*str* *or* *bytes*) – Absolute URL
* **user_agent** (*str* *or* *bytes*) – User agent
##### crawl_delay(user_agent: str | bytes) → float | None
Return the `Crawl-delay` directive for `user_agent` as a number
of seconds, or `None` if it is not set or the backend does not support
it.
> [!WARNING]
> Added in version 2.18.0.
#### DownloaderStats
#### *class* scrapy.downloadermiddlewares.stats.DownloaderStats
Middleware that stores stats of all requests, responses and exceptions that
pass through it.
To use this middleware you must enable the `DOWNLOADER_STATS`
setting.
#### UserAgentMiddleware
#### *class* scrapy.downloadermiddlewares.useragent.UserAgentMiddleware
Middleware that sets the `User-Agent` header.
The header value is taken from the `USER_AGENT` setting.
Source: /topics/spider-middleware.md
## Spider Middleware
The spider middleware is a framework of hooks into Scrapy’s spider processing
mechanism where you can plug custom functionality to process the responses that
are sent to topics-spiders for processing and to process the requests
and items that come out of a spider’s callback. See concepts for a
rundown of other alternatives.
### Activating a spider middleware
To activate a spider middleware component, add it to the
`SPIDER_MIDDLEWARES` setting, which is a dict whose keys are the
middleware class path and their values are the middleware orders.
Here’s an example:
```python
SPIDER_MIDDLEWARES = {
"myproject.middlewares.CustomSpiderMiddleware": 543,
}
```
The `SPIDER_MIDDLEWARES` setting is merged with the
`SPIDER_MIDDLEWARES_BASE` setting defined in Scrapy (and not meant to
be overridden) and then sorted by order to get the final sorted list of enabled
middlewares: the first middleware is the one closer to the engine and the last
is the one closer to the spider. In other words,
the `process_spider_input()`
method of each middleware will be invoked in increasing
middleware order (100, 200, 300, …), and the
`process_spider_output()` method
of each middleware will be invoked in decreasing order.
To decide which order to assign to your middleware see the
`SPIDER_MIDDLEWARES_BASE` setting and pick a value according to where
you want to insert the middleware. The order does matter because each
middleware performs a different action and your middleware could depend on some
previous (or subsequent) middleware being applied.
If you want to disable a builtin middleware (the ones defined in
`SPIDER_MIDDLEWARES_BASE`, and enabled by default) you must define it
in your project `SPIDER_MIDDLEWARES` setting and assign `None` as its
value. For example, if you want to disable the referer middleware:
```python
SPIDER_MIDDLEWARES = {
"scrapy.spidermiddlewares.referer.RefererMiddleware": None,
"myproject.middlewares.CustomRefererSpiderMiddleware": 700,
}
```
Finally, keep in mind that some middlewares may need to be enabled through a
particular setting. See each middleware documentation for more info.
### Writing your own spider middleware
Each spider middleware is a component that defines
one or more of these methods:
#### *class* scrapy.spidermiddlewares.SpiderMiddleware
##### *async* process_start(start: AsyncIterator[Any], /) → AsyncIterator[Any]
Iterate over the output of `start()` or that
of the `process_start()` method of an earlier spider middleware,
overriding it. For example:
```python
async def process_start(self, start):
async for item_or_request in start:
yield item_or_request
```
You may yield the same type of objects as `start()`.
To write spider middlewares that work on Scrapy versions lower than
2.13, define also a synchronous `process_start_requests()` method
that returns an iterable. For example:
```python
def process_start_requests(self, start, spider):
yield from start
```
##### process_spider_input(response)
This method is called for each response that goes through the spider
middleware and into the spider, for processing.
`process_spider_input()` should return `None` or raise an
exception.
If it returns `None`, Scrapy will continue processing this response,
executing all other middlewares until, finally, the response is handed
to the spider for processing.
If it raises an exception, Scrapy won’t bother calling any other spider
middleware `process_spider_input()` and will call the request
errback if there is one, otherwise it will start the `process_spider_exception()`
chain. The output of the errback is chained back in the other
direction for `process_spider_output()` to process it, or
`process_spider_exception()` if it raised an exception.
* **Parameters:**
**response** (`Response` object) – the response being processed
##### *async* process_spider_output(response, result)
This method is an asynchronous generator called with the
results from the spider after the spider has processed the response.
*result* is lazy: a generator callback runs as *result* is iterated, so
code that runs before that iteration runs before the callback body.
The number of objects yielded need not match the number received: drop
some, pass others through unchanged, or yield more than were received,
e.g. turning one item into several.
> [!NOTE]
> ### See also
>
> universal-spider-middleware.
* **Parameters:**
* **response** (`Response` object) – the response which generated this output from the
spider
* **result** (an asynchronous iterable of
`Request` objects and item objects) – the results from the spider
##### *async* process_spider_output_async(response, result)
Alternative name for `process_spider_output()` used when
implementing a universal spider middleware.
##### process_spider_exception(response, exception)
This method is called when a spider callback or a
`process_spider_output()` method (from a previous spider
middleware) raises an exception.
`process_spider_exception()` should return either `None` or an
iterable of `Request` or item
objects.
If it returns `None`, Scrapy will continue processing this exception,
executing any other `process_spider_exception()` in the following
middleware components, until no middleware components are left and the
exception reaches the engine (where it’s logged and discarded).
If it returns an iterable the `process_spider_output()` pipeline
kicks in, starting from the next spider middleware, and no other
`process_spider_exception()` will be called.
* **Parameters:**
* **response** (`Response` object) – the response being processed when the exception was
raised
* **exception** (`Exception` object) – the exception raised
#### Universal spider middlewares
In Scrapy 2.6.3 and lower, `process_spider_output()` must be a *synchronous*
generator.
To support those versions and higher Scrapy versions in the same middleware,
rename your asynchronous `process_spider_output()`
method to `process_spider_output_async()`, and define a
synchronous `process_spider_output()` method to be used by 2.6.3 and lower
versions.
For example:
```python
class UniversalSpiderMiddleware:
async def process_spider_output_async(self, response, result):
async for r in result:
# ... do something with r
yield r
def process_spider_output(self, response, result):
for r in result:
# ... do something with r
yield r
```
#### Base class for custom spider middlewares
Scrapy provides a base class for custom spider middlewares. It’s not required
to use it but it can help with simplifying middleware implementations.
#### *class* scrapy.spidermiddlewares.base.BaseSpiderMiddleware(crawler: Crawler)
Optional base class for spider middlewares.
> [!WARNING]
> Added in version 2.13.
This class provides helper methods for asynchronous
`process_spider_output()` and `process_start()` methods. Middlewares
that don’t have either of these methods don’t need to use this class.
You can override the
`get_processed_request()`
method to add processing code for requests and the
`get_processed_item()`
method to add processing code for items. These methods take a single
request or item from the spider output iterable and return a request or
item (the same or a new one), or `None` to remove this request or item
from the processing.
##### get_processed_request(request: Request, response: Response | None) → Request | None
Return a processed request from the spider output.
This method is called with a single request from `start()` or the
spider output. It should return the same or a different request, or
`None` to ignore it.
* **Parameters:**
* **request** (`Request` object) – the input request
* **response** (`Response` object or `None` for
start requests) – the response being processed
* **Returns:**
the processed request or `None`
##### get_processed_item(item: Any, response: Response | None) → Any
Return a processed item from the spider output.
This method is called with a single item from `start()` or the
spider output. It should return the same or a different item, or
`None` to ignore it.
* **Parameters:**
* **item** (*item object*) – the input item
* **response** (`Response` object or `None` for
start items) – the response being processed
* **Returns:**
the processed item or `None`
### Built-in spider middleware reference
This page describes all spider middleware components that come with Scrapy. For
information on how to use them and how to write your own spider middleware, see
the spider middleware usage guide.
For a list of the components enabled by default (and their orders) see the
`SPIDER_MIDDLEWARES_BASE` setting.
#### DepthMiddleware
#### *class* scrapy.spidermiddlewares.depth.DepthMiddleware(maxdepth: int, stats: StatsCollector, verbose_stats: bool = False, prio: int = 1)
Track the depth of each request within the site being scraped, setting
`request.meta["depth"]` to 0 when there is no value previously set
(usually just the first request) and incrementing it by 1 otherwise.
It can be used to limit the maximum depth to scrape, control request
priority based on their depth, and things like that, through the
`DEPTH_LIMIT`, `DEPTH_STATS_VERBOSE` and
`DEPTH_PRIORITY` settings.
##### depth_reset
> [!WARNING]
> Added in version 2.18.0.
`meta` key that, set to `True`, gives a request
depth 0 instead of the depth of its source response plus 1, e.g. to keep
`DEPTH_LIMIT` from applying across a domain change.
#### HttpErrorMiddleware
#### *class* scrapy.spidermiddlewares.httperror.HttpErrorMiddleware
Filter out unsuccessful (erroneous) HTTP responses so that spiders don’t
have to deal with them, which (most of the time) imposes an overhead,
consumes more resources, and makes the spider logic more complex.
According to the [HTTP standard](https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html), successful responses are those whose
status codes are in the 200-300 range.
If you still want to process response codes outside that range, you can
specify which response codes the spider is able to handle using the
`handle_httpstatus_list` spider attribute or
`HTTPERROR_ALLOWED_CODES` setting.
For example, if you want your spider to handle 404 responses you can do
this:
```python
from scrapy.spiders import CrawlSpider
class MySpider(CrawlSpider):
handle_httpstatus_list = [404]
```
The `handle_httpstatus_list` key of `Request.meta` can also be used to specify which response codes to
allow on a per-request basis. You can also set the meta key `handle_httpstatus_all`
to `True` if you want to allow any response code for a request, and `False` to
disable the effects of the `handle_httpstatus_all` key.
Keep in mind, however, that it’s usually a bad idea to handle non-200
responses, unless you really know what you’re doing.
For more information see: [HTTP Status Code Definitions](https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html).
##### HttpErrorMiddleware settings
###### HTTPERROR_ALLOWED_CODES
Default: `[]`
Pass all responses with non-200 status codes contained in this list.
###### HTTPERROR_ALLOW_ALL
Default: `False`
Pass all responses, regardless of its status code.
#### MetaCopyDetectionMiddleware
#### *class* scrapy.spidermiddlewares.metacopy.MetaCopyDetectionMiddleware
Warns when a spider yields a request that contains internal meta keys which
should not be copied from `response.meta`
into new requests. See `meta` to learn why.
Only 1 warning is emitted per crawl.
##### MetaCopyDetectionMiddleware settings
###### META_COPY_WARN_SKIP_KEYS
Default: `[]`
A list of internal meta key names to exclude from the internal-keys check.
Use this when you intentionally copy one of the monitored keys and want to
suppress the resulting warning without disabling the middleware entirely.
#### RefererMiddleware
#### *class* scrapy.spidermiddlewares.referer.RefererMiddleware
Populates Request `Referer` header, based on the URL of the Response which
generated it.
##### RefererMiddleware settings
###### REFERER_ENABLED
Default: `True`
Whether to enable referer middleware.
###### REFERRER_POLICY
Default: `'scrapy.spidermiddlewares.referer.DefaultReferrerPolicy'`
[Referrer Policy](https://www.w3.org/TR/referrer-policy) to apply when populating Request “Referer” header.
> [!NOTE]
> You can also set the Referrer Policy per request,
> using the special `"referrer_policy"` Request.meta key,
> with the same acceptable values as for the `REFERRER_POLICY` setting.
> [!NOTE]
> ### See also
>
> security-credential-leakage
###### Acceptable values for REFERRER_POLICY
- either a path to a `scrapy.spidermiddlewares.referer.ReferrerPolicy`
subclass — a custom policy or one of the built-in ones (see classes below),
- or one or more comma-separated standard W3C-defined string values,
- or the special `"scrapy-default"`.
| String value | Class name (as a string) |
|-----------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------|
| `"scrapy-default"` (default) | `scrapy.spidermiddlewares.referer.DefaultReferrerPolicy` |
| [“no-referrer”](https://www.w3.org/TR/referrer-policy/#referrer-policy-no-referrer) | `scrapy.spidermiddlewares.referer.NoReferrerPolicy` |
| [“no-referrer-when-downgrade”](https://www.w3.org/TR/referrer-policy/#referrer-policy-no-referrer-when-downgrade) | `scrapy.spidermiddlewares.referer.NoReferrerWhenDowngradePolicy` |
| [“same-origin”](https://www.w3.org/TR/referrer-policy/#referrer-policy-same-origin) | `scrapy.spidermiddlewares.referer.SameOriginPolicy` |
| [“origin”](https://www.w3.org/TR/referrer-policy/#referrer-policy-origin) | `scrapy.spidermiddlewares.referer.OriginPolicy` |
| [“strict-origin”](https://www.w3.org/TR/referrer-policy/#referrer-policy-strict-origin) | `scrapy.spidermiddlewares.referer.StrictOriginPolicy` |
| [“origin-when-cross-origin”](https://www.w3.org/TR/referrer-policy/#referrer-policy-origin-when-cross-origin) | `scrapy.spidermiddlewares.referer.OriginWhenCrossOriginPolicy` |
| [“strict-origin-when-cross-origin”](https://www.w3.org/TR/referrer-policy/#referrer-policy-strict-origin-when-cross-origin) | `scrapy.spidermiddlewares.referer.StrictOriginWhenCrossOriginPolicy` |
| [“unsafe-url”](https://www.w3.org/TR/referrer-policy/#referrer-policy-unsafe-url) | `scrapy.spidermiddlewares.referer.UnsafeUrlPolicy` |
#### *class* scrapy.spidermiddlewares.referer.ReferrerPolicy
Abstract base class for referrer policies.
#### *class* scrapy.spidermiddlewares.referer.DefaultReferrerPolicy
A variant of “no-referrer-when-downgrade”,
with the addition that “Referer” is not sent if the parent request was
using `file://` or `s3://` scheme.
> [!WARNING]
> Scrapy’s default referrer policy — just like [“no-referrer-when-downgrade”](https://www.w3.org/TR/referrer-policy/#referrer-policy-no-referrer-when-downgrade),
> the W3C-recommended value for browsers — will send a non-empty
> “Referer” header from any `http(s)://` to any `https://` URL,
> even if the domain is different.
>
> [“same-origin”](https://www.w3.org/TR/referrer-policy/#referrer-policy-same-origin) may be a better choice if you want to remove referrer
> information for cross-domain requests.
#### *class* scrapy.spidermiddlewares.referer.NoReferrerPolicy
[https://www.w3.org/TR/referrer-policy/#referrer-policy-no-referrer](https://www.w3.org/TR/referrer-policy/#referrer-policy-no-referrer)
The simplest policy is “no-referrer”, which specifies that no referrer information
is to be sent along with requests made from a particular request client to any origin.
The header will be omitted entirely.
#### *class* scrapy.spidermiddlewares.referer.NoReferrerWhenDowngradePolicy
[https://www.w3.org/TR/referrer-policy/#referrer-policy-no-referrer-when-downgrade](https://www.w3.org/TR/referrer-policy/#referrer-policy-no-referrer-when-downgrade)
The “no-referrer-when-downgrade” policy sends a full URL along with requests
from a TLS-protected environment settings object to a potentially trustworthy URL,
and requests from clients which are not TLS-protected to any origin.
Requests from TLS-protected clients to non-potentially trustworthy URLs,
on the other hand, will contain no referrer information.
A Referer HTTP header will not be sent.
This is a user agent’s default behavior, if no policy is otherwise specified.
> [!NOTE]
> “no-referrer-when-downgrade” policy is the W3C-recommended default,
> and is used by major web browsers.
>
> However, it is NOT Scrapy’s default referrer policy (see `DefaultReferrerPolicy`).
#### *class* scrapy.spidermiddlewares.referer.SameOriginPolicy
[https://www.w3.org/TR/referrer-policy/#referrer-policy-same-origin](https://www.w3.org/TR/referrer-policy/#referrer-policy-same-origin)
The “same-origin” policy specifies that a full URL, stripped for use as a referrer,
is sent as referrer information when making same-origin requests from a particular request client.
Cross-origin requests, on the other hand, will contain no referrer information.
A Referer HTTP header will not be sent.
#### *class* scrapy.spidermiddlewares.referer.OriginPolicy
[https://www.w3.org/TR/referrer-policy/#referrer-policy-origin](https://www.w3.org/TR/referrer-policy/#referrer-policy-origin)
The “origin” policy specifies that only the ASCII serialization
of the origin of the request client is sent as referrer information
when making both same-origin requests and cross-origin requests
from a particular request client.
#### *class* scrapy.spidermiddlewares.referer.StrictOriginPolicy
[https://www.w3.org/TR/referrer-policy/#referrer-policy-strict-origin](https://www.w3.org/TR/referrer-policy/#referrer-policy-strict-origin)
The “strict-origin” policy sends the ASCII serialization
of the origin of the request client when making requests:
- from a TLS-protected environment settings object to a potentially trustworthy URL, and
- from non-TLS-protected environment settings objects to any origin.
Requests from TLS-protected request clients to non- potentially trustworthy URLs,
on the other hand, will contain no referrer information.
A Referer HTTP header will not be sent.
#### *class* scrapy.spidermiddlewares.referer.OriginWhenCrossOriginPolicy
[https://www.w3.org/TR/referrer-policy/#referrer-policy-origin-when-cross-origin](https://www.w3.org/TR/referrer-policy/#referrer-policy-origin-when-cross-origin)
The “origin-when-cross-origin” policy specifies that a full URL,
stripped for use as a referrer, is sent as referrer information
when making same-origin requests from a particular request client,
and only the ASCII serialization of the origin of the request client
is sent as referrer information when making cross-origin requests
from a particular request client.
#### *class* scrapy.spidermiddlewares.referer.StrictOriginWhenCrossOriginPolicy
[https://www.w3.org/TR/referrer-policy/#referrer-policy-strict-origin-when-cross-origin](https://www.w3.org/TR/referrer-policy/#referrer-policy-strict-origin-when-cross-origin)
The “strict-origin-when-cross-origin” policy specifies that a full URL,
stripped for use as a referrer, is sent as referrer information
when making same-origin requests from a particular request client,
and only the ASCII serialization of the origin of the request client
when making cross-origin requests:
- from a TLS-protected environment settings object to a potentially trustworthy URL, and
- from non-TLS-protected environment settings objects to any origin.
Requests from TLS-protected clients to non- potentially trustworthy URLs,
on the other hand, will contain no referrer information.
A Referer HTTP header will not be sent.
#### *class* scrapy.spidermiddlewares.referer.UnsafeUrlPolicy
[https://www.w3.org/TR/referrer-policy/#referrer-policy-unsafe-url](https://www.w3.org/TR/referrer-policy/#referrer-policy-unsafe-url)
The “unsafe-url” policy specifies that a full URL, stripped for use as a referrer,
is sent along with both cross-origin requests
and same-origin requests made from a particular request client.
Note: The policy’s name doesn’t lie; it is unsafe.
This policy will leak origins and paths from TLS-protected resources
to insecure origins.
Carefully consider the impact of setting such a policy for potentially sensitive documents.
> [!WARNING]
> “unsafe-url” policy is NOT recommended.
###### REFERRER_POLICIES
> [!WARNING]
> Added in version 2.14.2.
Default: `{}`
A dictionary mapping policy names to import paths of
`scrapy.spidermiddlewares.referer.ReferrerPolicy` subclasses, or
`None` to disable support for a given policy name.
This allows overriding the policies triggered by the `Referrer-Policy`
response header.
Use `""` to override the policy for responses with [no referrer policy](https://www.w3.org/TR/referrer-policy/#referrer-policy-empty-string).
#### StartSpiderMiddleware
#### *class* scrapy.spidermiddlewares.start.StartSpiderMiddleware(crawler: Crawler)
Set `is_start_request`.
##### is_start_request
`meta` key that is set to `True` in start
requests, allowing you to tell start requests apart from
other requests, e.g. in downloader middlewares.
#### UrlLengthMiddleware
#### *class* scrapy.spidermiddlewares.urllength.UrlLengthMiddleware
Filters out requests with URLs longer than URLLENGTH_LIMIT
The `UrlLengthMiddleware` can be configured through the following
settings (see the settings documentation for more info):
> * `URLLENGTH_LIMIT` - The maximum URL length to allow for crawled URLs.
Source: /topics/extensions.md
## Extensions
Extensions are components that allow inserting your
own custom functionality into Scrapy.
Unlike other components, extensions do not have a specific role in Scrapy. They
are “wildcard” components that can be used for anything that does not fit the
role of any other type of component, such as collecting stats or enforcing a
memory limit; see concepts for a rundown of other alternatives.
### Loading and activating extensions
Extensions are loaded at startup by creating a single instance of the extension
class per spider being run.
To enable an extension, add it to the `EXTENSIONS` setting. For
example:
```python
EXTENSIONS = {
"scrapy.extensions.corestats.CoreStats": 500,
"scrapy.extensions.telnet.TelnetConsole": 500,
}
```
`EXTENSIONS` is merged with `EXTENSIONS_BASE` (not meant to
be overridden), and the priorities in the resulting value determine the
*loading* order.
As extensions typically do not depend on each other, their loading order is
irrelevant in most cases. This is why the `EXTENSIONS_BASE` setting
defines all extensions with the same order (`0`). However, you may need to
carefully use priorities if you add an extension that depends on other
extensions being already loaded.
### Writing your own extension
Each extension is a component.
Typically, extensions connect to signals and perform
tasks triggered by them.
#### Sample extension
Here we will implement a simple extension to illustrate the concepts described
in the previous section. This extension will log a message every time:
* a spider is opened
* a spider is closed
* a specific number of items are scraped
The extension will be enabled through the `MYEXT_ENABLED` setting and the
number of items will be specified through the `MYEXT_ITEMCOUNT` setting.
Here is the code of such extension:
```python
import logging
from scrapy import signals
from scrapy.exceptions import NotConfigured
logger = logging.getLogger(__name__)
class SpiderOpenCloseLogging:
def __init__(self, item_count):
self.item_count = item_count
self.items_scraped = 0
@classmethod
def from_crawler(cls, crawler):
# first check if the extension should be enabled and raise
# NotConfigured otherwise
if not crawler.settings.getbool("MYEXT_ENABLED"):
raise NotConfigured
# get the number of items from settings
item_count = crawler.settings.getint("MYEXT_ITEMCOUNT", 1000)
# instantiate the extension object
ext = cls(item_count)
# connect the extension object to signals
crawler.signals.connect(ext.spider_opened, signal=signals.spider_opened)
crawler.signals.connect(ext.spider_closed, signal=signals.spider_closed)
crawler.signals.connect(ext.item_scraped, signal=signals.item_scraped)
# return the extension object
return ext
def spider_opened(self, spider):
logger.info("opened spider %s", spider.name)
def spider_closed(self, spider):
logger.info("closed spider %s", spider.name)
def item_scraped(self, item, spider):
self.items_scraped += 1
if self.items_scraped % self.item_count == 0:
logger.info("scraped %d items", self.items_scraped)
```
### Built-in extensions reference
#### General purpose extensions
##### Log Stats extension
#### *class* scrapy.extensions.logstats.LogStats
Log basic stats like crawled pages and scraped items.
##### Core Stats extension
#### *class* scrapy.extensions.corestats.CoreStats
Enable the collection of core statistics, provided the stats collection is
enabled (see topics-stats).
The following stats are collected: `elapsed_time_seconds`,
`finish_reason`, `finish_time`, `item_dropped_count`,
`item_dropped_reasons_count/{exception}`, `item_scraped_count`,
`response_received_count`, `start_time`.
##### Log Count extension
#### *class* scrapy.extensions.logcount.LogCount(crawler: Crawler)
Install a log handler that counts log messages by level.
The handler installed is `scrapy.utils.log.LogCounterHandler`.
The counts are stored in the `log_count/{level}` stat.
> [!WARNING]
> Added in version 2.14.
##### Memory usage extension
#### *class* scrapy.extensions.memusage.MemoryUsage
> [!NOTE]
> This extension does not work in Windows.
Monitors the memory used by the Scrapy process that runs the spider and:
1. sends a `memusage_warning_reached` signal when it exceeds
`MEMUSAGE_WARNING_MB`
2. closes the spider with the `"memusage_exceeded"` reason when it exceeds
`MEMUSAGE_LIMIT_MB`
This extension is enabled by the `MEMUSAGE_ENABLED` setting and
can be configured with the following settings:
* `MEMUSAGE_LIMIT_MB`
* `MEMUSAGE_WARNING_MB`
* `MEMUSAGE_CHECK_INTERVAL_SECONDS`
##### Memory debugger extension
#### *class* scrapy.extensions.memdebug.MemoryDebugger
An extension for debugging memory usage. It collects information about:
* objects uncollected by the Python garbage collector
* objects left alive that shouldn’t. For more info, see topics-leaks-trackrefs
To enable this extension, turn on the `MEMDEBUG_ENABLED` setting. The
info will be stored in the `memdebug/gc_garbage_count` and
`memdebug/live_refs/{cls}` stats.
##### Spider state extension
#### *class* scrapy.extensions.spiderstate.SpiderState
Manages spider state data by loading it before a crawl and saving it after.
Give a value to the `JOBDIR` setting to enable this extension.
When enabled, this extension manages the `state`
attribute of your `Spider` instance:
- When your spider closes (`spider_closed`), the contents of its
`state` attribute are serialized into a file named
`spider.state` in the `JOBDIR` folder.
- When your spider opens (`spider_opened`), if a previously-generated
`spider.state` file exists in the `JOBDIR` folder, it is loaded
into the `state` attribute.
For an example, see topics-keeping-persistent-state-between-batches.
##### Close spider extension
#### *class* scrapy.extensions.closespider.CloseSpider
Closes a spider automatically when some conditions are met, using a specific
closing reason for each condition.
The conditions for closing a spider can be configured through the following
settings:
* `CLOSESPIDER_TIMEOUT`
* `CLOSESPIDER_TIMEOUT_NO_ITEM`
* `CLOSESPIDER_ITEMCOUNT`
* `CLOSESPIDER_PAGECOUNT`
* `CLOSESPIDER_PAGECOUNT_NO_ITEM`
* `CLOSESPIDER_ERRORCOUNT`
> [!NOTE]
> When a certain closing condition is met, requests which are
> currently in the downloader queue (up to `CONCURRENT_REQUESTS`
> requests) are still processed.
###### CLOSESPIDER_TIMEOUT
Default: `0.0`
If the spider remains open for more than this number of seconds, it will be
automatically closed with the reason `closespider_timeout`. If zero (or non
set), spiders won’t be closed by timeout.
###### CLOSESPIDER_TIMEOUT_NO_ITEM
Default: `0`
An integer which specifies a number of seconds. If the spider has not produced
any items in the last number of seconds, it will be closed with the reason
`closespider_timeout_no_item`. If zero (or non set), spiders won’t be closed
regardless if it hasn’t produced any items.
###### CLOSESPIDER_ITEMCOUNT
Default: `0`
An integer which specifies a number of items. If the spider scrapes more than
that amount and those items are passed by the item pipeline, the
spider will be closed with the reason `closespider_itemcount`.
If zero (or non set), spiders won’t be closed by number of passed items.
###### CLOSESPIDER_PAGECOUNT
Default: `0`
An integer which specifies the maximum number of responses to crawl. If the spider
crawls more than that, the spider will be closed with the reason
`closespider_pagecount`. If zero (or non set), spiders won’t be closed by
number of crawled responses.
###### CLOSESPIDER_PAGECOUNT_NO_ITEM
Default: `0`
An integer which specifies the maximum number of consecutive responses to crawl
without items scraped. If the spider crawls more consecutive responses than that
and no items are scraped in the meantime, the spider will be closed with the
reason `closespider_pagecount_no_item`. If zero (or not set), spiders won’t be
closed by number of crawled responses with no items.
###### CLOSESPIDER_ERRORCOUNT
Default: `0`
An integer which specifies the maximum number of errors to receive before
closing the spider. If the spider generates more than that number of errors,
it will be closed with the reason `closespider_errorcount`. If zero (or non
set), spiders won’t be closed by number of errors.
##### Periodic log extension
#### *class* scrapy.extensions.periodic_log.PeriodicLog
This extension periodically logs rich stat data as a JSON object:
```
2023-08-04 02:30:57 [scrapy.extensions.logstats] INFO: Crawled 976 pages (at 162 pages/min), scraped 925 items (at 161 items/min)
2023-08-04 02:30:57 [scrapy.extensions.periodic_log] INFO: {
"delta": {
"downloader/request_bytes": 55582,
"downloader/request_count": 162,
"downloader/request_method_count/GET": 162,
"downloader/response_bytes": 618133,
"downloader/response_count": 162,
"downloader/response_status_count/200": 162,
"item_scraped_count": 161
},
"stats": {
"downloader/request_bytes": 338243,
"downloader/request_count": 992,
"downloader/request_method_count/GET": 992,
"downloader/response_bytes": 3836736,
"downloader/response_count": 976,
"downloader/response_status_count/200": 976,
"item_scraped_count": 925,
"log_count/INFO": 21,
"log_count/WARNING": 1,
"scheduler/dequeued": 992,
"scheduler/dequeued/memory": 992,
"scheduler/enqueued": 1050,
"scheduler/enqueued/memory": 1050
},
"time": {
"elapsed": 360.008903,
"log_interval": 60.0,
"log_interval_real": 60.006694,
"start_time": "2023-08-03T23:24:57.148903+00:00",
"utcnow": "2023-08-03T23:30:57.157806+00:00"
}
}
```
This extension logs the following configurable sections:
- `"delta"` shows how some numeric stats have changed since the last stats
log message.
The `PERIODIC_LOG_DELTA` setting determines the target stats. They
must have `int` or `float` values.
- `"stats"` shows the current value of some stats.
The `PERIODIC_LOG_STATS` setting determines the target stats.
- `"time"` shows detailed timing data.
The `PERIODIC_LOG_TIMING_ENABLED` setting determines whether or
not to show this section.
This extension logs data at the start, then on a fixed time interval
configurable through the `LOGSTATS_INTERVAL` setting, and finally
right before the crawl ends.
Example extension configuration:
```python
custom_settings = {
"LOG_LEVEL": "INFO",
"PERIODIC_LOG_STATS": {
"include": ["downloader/", "scheduler/", "log_count/", "item_scraped_count"],
},
"PERIODIC_LOG_DELTA": {"include": ["downloader/"]},
"PERIODIC_LOG_TIMING_ENABLED": True,
"EXTENSIONS": {
"scrapy.extensions.periodic_log.PeriodicLog": 0,
},
}
```
###### PERIODIC_LOG_DELTA
Default: `None`
* `"PERIODIC_LOG_DELTA": True` - show deltas for all `int` and `float` stat values.
* `"PERIODIC_LOG_DELTA": {"include": ["downloader/", "scheduler/"]}` - show deltas for stats with names containing any configured substring.
* `"PERIODIC_LOG_DELTA": {"exclude": ["downloader/"]}` - show deltas for all stats with names not containing any configured substring.
###### PERIODIC_LOG_STATS
Default: `None`
* `"PERIODIC_LOG_STATS": True` - show the current value of all stats.
* `"PERIODIC_LOG_STATS": {"include": ["downloader/", "scheduler/"]}` - show current values for stats with names containing any configured substring.
* `"PERIODIC_LOG_STATS": {"exclude": ["downloader/"]}` - show current values for all stats with names not containing any configured substring.
###### PERIODIC_LOG_TIMING_ENABLED
Default: `False`
`True` enables logging of timing data (i.e. the `"time"` section).
#### Debugging extensions
##### Stack trace dump extension
#### *class* scrapy.extensions.debug.StackTraceDump
Dumps information about the running process when a [SIGQUIT](https://en.wikipedia.org/wiki/SIGQUIT) or [SIGUSR2](https://en.wikipedia.org/wiki/SIGUSR1_and_SIGUSR2)
signal is received. The information dumped is the following:
1. engine status (using `scrapy.utils.engine.get_engine_status()`)
2. live references (see topics-leaks-trackrefs)
3. stack trace of all threads
After the stack trace and engine status is dumped, the Scrapy process continues
running normally.
This extension only works on POSIX-compliant platforms (i.e. not Windows),
because the [SIGQUIT](https://en.wikipedia.org/wiki/SIGQUIT) and [SIGUSR2](https://en.wikipedia.org/wiki/SIGUSR1_and_SIGUSR2) signals are not available on Windows.
There are at least two ways to send Scrapy the [SIGQUIT](https://en.wikipedia.org/wiki/SIGQUIT) signal:
1. By pressing Ctrl-while a Scrapy process is running (Linux only?)
2. By running this command (assuming `<pid>` is the process id of the Scrapy
process):
```
kill -QUIT <pid>
```
##### Debugger extension
#### *class* scrapy.extensions.debug.Debugger
Invokes a Python debugger inside a running Scrapy process when a [SIGUSR2](https://en.wikipedia.org/wiki/SIGUSR1_and_SIGUSR2)
signal is received. After the debugger is exited, the Scrapy process continues
running normally.
This extension only works on POSIX-compliant platforms (i.e. not Windows).
##### Telnet console extension
#### *class* scrapy.extensions.telnet.TelnetConsole
Provides a telnet console for getting into a Python interpreter inside the
currently running Scrapy process, which can be very useful for debugging.
The telnet console must be enabled by the `TELNETCONSOLE_ENABLED`
setting, and the server will listen in the port specified in
`TELNETCONSOLE_PORT`.
##### Remote control extension
> [!WARNING]
> Added in version 2.19.0.
#### *class* scrapy.extensions.remote_control.RemoteControl(crawler: Crawler)
Provides an HTTP server that can run Python code passed to it in HTTP requests
and return the output in responses.
The code runs inside the Scrapy process and has access to the
`Crawler` instance in the `crawler` variable and
to a persistent dictionary in the `stash` variable.
This extension can be disabled by setting the
`REMOTE_CONTROL_ENABLED` setting to `False`. It requires
asyncio support and will be disabled without it.
The HTTP server listens on a random port on `localhost` and requires a
`Bearer` token for authentication. The token and port are written to a
job file in the user’s profile directory so that other processes can
discover and use them to connect to the server.
Available endpoints:
- `/status`: expects a `GET` request and returns a JSON object with the
following keys:
> - `pid` (number): the process ID of the Scrapy process.
> - `spider` (string): the name of the currently running spider.
> - `project` (string): the name of the Scrapy project.
> - `scrapy_version` (string): the version of Scrapy.
> - `start_time` (number or null): the start time of the Scrapy process
> as a UNIX timestamp.
- `/execute`: expects a `POST` request with a JSON object containing
the following keys:
> - `code` (string): Python code to execute.
> - `timeout_sec` (number, optional): the maximum number of seconds
> to allow the code to run. This will cancel async operations but not
> sync ones.
> The response is a JSON object with the following keys:
> - `status` (string): one of `"ok"` (the code was executed
> successfully), `"compile_error"` (the code could not be compiled),
> `"error"` (the code raised an exception), or `"timeout"` (the
> code couldn’t finish within the given time).
> - `output` (string): the output produced by `print()` calls in the
> code.
> - `traceback` (string or null): the traceback if an exception was
> raised when compiling or executing the code.
> - `elapsed_sec` (number): the number of seconds the code took to run.
> - `output_truncated` (boolean, optional): whether `output` was
> truncated (omitted if `false`).
> - `traceback_truncated` (boolean, optional): whether `traceback`
> was truncated (omitted if `false`).
This extension can be used with any HTTP client, but it is intended to be
used by the Scrapy MCP server (see using-mcp-server).
###### REMOTE_CONTROL_ENABLED
Default: `True`
Whether to enable the `RemoteControl` extension.
###### REMOTE_CONTROL_JOBS_DIR
Default: `None`
The directory for storing `RemoteControl` job files. When this is set to
`None`, a `scrapy/job_files` subdirectory in
`platformdirs.user_state_dir()` is used.
As job files contain authentication tokens necessary to connect to Scrapy
processes, this directory should not be exposed to untrusted environments.
###### REMOTE_CONTROL_TIMEOUT_DEFAULT
Default: `30.0`
The default timeout in seconds for running a single code snippet sent to the
`RemoteControl` `/execute` endpoint. You can override it for a single
request via the `timeout_sec` request field.
###### REMOTE_CONTROL_TIMEOUT_MAX
Default: `600.0`
The maximum allowed value for the `timeout_sec` field of
`RemoteControl` `/execute` endpoint requests. Higher values will be
clamped to this value.
###### REMOTE_CONTROL_OUTPUT_MAX_BYTES
Default: `65536`
The maximum size of the `output` field in responses of `RemoteControl`
`/execute` endpoint requests. Longer ones will be truncated.
###### REMOTE_CONTROL_TRACEBACK_MAX_BYTES
Default: `16384`
The maximum size of the `traceback` field in responses of
`RemoteControl` `/execute` endpoint requests. Longer ones will be
truncated.
Source: /topics/signals.md
## Signals
Scrapy uses signals extensively to notify when certain events occur. You can
catch some of those signals in your Scrapy project (using an extension, for example) to perform additional tasks or extend Scrapy
to add functionality not provided out of the box.
Even though signals provide several arguments, the handlers that catch them
don’t need to accept all of them - the signal dispatching mechanism will only
deliver the arguments that the handler receives.
You can connect to signals (or send your own) through the
topics-api-signals.
Here is a simple example showing how you can catch signals and perform some action:
```python
from scrapy import signals
from scrapy import Spider
class DmozSpider(Spider):
name = "dmoz"
allowed_domains = ["dmoz.org"]
start_urls = [
"http://www.dmoz.org/Computers/Programming/Languages/Python/Books/",
"http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/",
]
@classmethod
def from_crawler(cls, crawler, *args, **kwargs):
spider = super().from_crawler(crawler, *args, **kwargs)
crawler.signals.connect(spider.spider_closed, signal=signals.spider_closed)
return spider
def spider_closed(self, spider):
spider.logger.info("Spider closed: %s", spider.name)
def parse(self, response):
pass
```
### Handler order
The order in which the handlers of a signal run is undefined, and
asynchronous handlers run concurrently. If two actions
must happen in a given order, run both from a single handler, in that order.
### Asynchronous signal handlers
Some signals support returning `Deferred`
or awaitable objects from their handlers, allowing
you to run asynchronous code that does not block Scrapy. If a signal
handler returns one of these objects, Scrapy waits for that asynchronous
operation to finish.
Let’s take an example using coroutines:
```python
import json
import scrapy
import treq
class SignalSpider(scrapy.Spider):
name = "signals"
start_urls = ["https://quotes.toscrape.com/page/1/"]
@classmethod
def from_crawler(cls, crawler, *args, **kwargs):
spider = super().from_crawler(crawler, *args, **kwargs)
crawler.signals.connect(spider.item_scraped, signal=signals.item_scraped)
return spider
async def item_scraped(self, item):
# Send the scraped item to the server
response = await treq.post(
"http://example.com/post",
json.dumps(item).encode("ascii"),
headers={b"Content-Type": [b"application/json"]},
)
return response
def parse(self, response):
for quote in response.css("div.quote"):
yield {
"text": quote.css("span.text::text").get(),
"author": quote.css("small.author::text").get(),
"tags": quote.css("div.tags a.tag::text").getall(),
}
```
See the topics-signals-ref below to know which signals support
`Deferred` and awaitable objects.
### Built-in signals reference
Here’s the list of Scrapy built-in signals and their meaning.
#### Engine signals
##### engine_started
#### scrapy.signals.engine_started()
Sent when the Scrapy engine has started crawling.
This signal supports asynchronous handlers.
> [!NOTE]
> This signal may be fired *after* the `spider_opened` signal,
> depending on how the spider was started. So **don’t** rely on this signal
> getting fired before `spider_opened`.
##### engine_stopped
#### scrapy.signals.engine_stopped()
Sent when the Scrapy engine is stopped (for example, when a crawling
process has finished).
This signal supports asynchronous handlers.
##### scheduler_empty
#### scrapy.signals.scheduler_empty()
Sent whenever the engine asks for a pending request from the
scheduler (i.e. calls its
`next_request()` method) and the
scheduler returns none.
See start-requests-lazy for an example.
> [!WARNING]
> Only wait for this signal from
> `start()`. While no request can be sent, e.g. while
> the responses being parsed exceed
> `SCRAPER_SLOT_MAX_ACTIVE_SIZE`, the engine does not ask the
> scheduler for requests, and hence this signal is not sent. So waiting
> for it from a callback can hang the crawl,
> because the response being parsed is itself one of the responses that
> may be blocking requests.
This signal does not support asynchronous handlers.
#### Item signals
> [!NOTE]
> As at max `CONCURRENT_ITEMS` items are processed in
> parallel, many deferreds are fired together using
> `DeferredList`. Hence the next
> batch waits for the `DeferredList`
> to fire and then runs the respective item signal handler for
> the next batch of scraped items.
##### item_scraped
#### scrapy.signals.item_scraped(item, response, spider)
Sent when an item has been scraped, after it has passed all the
topics-item-pipeline stages (without being dropped).
This signal supports asynchronous handlers.
* **Parameters:**
* **item** (item object) – the scraped item
* **spider** (`Spider` object) – the spider which scraped the item
* **response** (`Response` | `None`) – the response from where the item was scraped, or `None`
if it was yielded from `start()`.
##### item_dropped
#### scrapy.signals.item_dropped(item, response, exception, spider)
Sent after an item has been dropped from the topics-item-pipeline
when some stage raised a `DropItem` exception.
This signal supports asynchronous handlers.
* **Parameters:**
* **item** (item object) – the item dropped from the topics-item-pipeline
* **spider** (`Spider` object) – the spider which scraped the item
* **response** (`Response` | `None`) – the response from where the item was dropped, or `None`
if it was yielded from `start()`.
* **exception** (`DropItem` exception) – the exception (which must be a
`DropItem` subclass) which caused the item
to be dropped
##### item_error
#### scrapy.signals.item_error(item, response, spider, failure)
Sent when a topics-item-pipeline generates an error (i.e. raises
an exception), except `DropItem` exception.
This signal supports asynchronous handlers.
* **Parameters:**
* **item** (item object) – the item that caused the error in the topics-item-pipeline
* **response** (`Response` | `None`) – the response being processed when the exception was
raised, or `None` if it was yielded from
`start()`.
* **spider** (`Spider` object) – the spider which raised the exception
* **failure** (*twisted.python.failure.Failure*) – the exception raised
#### Spider signals
##### spider_closed
#### scrapy.signals.spider_closed(spider, reason)
Sent after a spider has been closed. This can be used to release per-spider
resources reserved on `spider_opened`.
This signal supports asynchronous handlers.
* **Parameters:**
* **spider** (`Spider` object) – the spider which has been closed
* **reason** (*str*) – a string which describes the reason why the spider was closed. If
it was closed because the spider has completed scraping, the reason
is `'finished'`. Otherwise, if the spider was manually closed by
calling the `close_spider` engine method, then the reason is the one
passed in the `reason` argument of that method (which defaults to
`'cancelled'`). If the engine was shutdown (for example, by hitting
Ctrl-C to stop it) the reason will be `'shutdown'`.
##### spider_opened
#### scrapy.signals.spider_opened(spider)
Sent after a spider has been opened for crawling. This is typically used to
reserve per-spider resources, but can be used for any task that needs to be
performed when a spider is opened.
> [!WARNING]
> Changed in version 2.18.0: Added support for `CloseSpider`.
You may raise a `CloseSpider` exception to close the
spider before it starts crawling, e.g. if a resource that the spider needs
is unavailable.
This signal supports asynchronous handlers.
* **Parameters:**
**spider** (`Spider` object) – the spider which has been opened
##### spider_idle
#### scrapy.signals.spider_idle(spider)
Sent when a spider has gone idle, which means the spider has no further:
> * requests waiting to be downloaded
> * requests scheduled
> * items being processed in the item pipeline
If the idle state persists after all handlers of this signal have finished,
the engine starts closing the spider. After the spider has finished
closing, the `spider_closed` signal is sent.
You may raise a `DontCloseSpider` exception to
prevent the spider from being closed.
Alternatively, you may raise a `CloseSpider`
exception to provide a custom spider closing reason. An
idle handler is the perfect place to put some code that assesses
the final spider results and update the final closing reason
accordingly (e.g. setting it to ‘too_few_results’ instead of
‘finished’).
This signal does not support asynchronous handlers.
* **Parameters:**
**spider** (`Spider` object) – the spider which has gone idle
> [!NOTE]
> Scheduling some requests in your `spider_idle` handler does
> **not** guarantee that it can prevent the spider from being closed,
> although it sometimes can. That’s because the spider may still remain idle
> if all the scheduled requests are rejected by the scheduler (e.g. filtered
> due to duplication).
##### spider_error
#### scrapy.signals.spider_error(failure, response, spider)
Sent when a spider callback or the `start()` method of a
spider generates an error (i.e. raises an exception).
> [!WARNING]
> Changed in version 2.18.0: Exceptions from `start()` are also reported, see
> start-error.
This signal does not support asynchronous handlers.
* **Parameters:**
* **failure** (*twisted.python.failure.Failure*) – the exception raised
* **response** (`Response` | `None`) – the response being processed when the exception was
raised, or `None` if the exception came from
`start()`.
* **spider** (`Spider` object) – the spider which raised the exception
##### feed_slot_closed
#### scrapy.signals.feed_slot_closed(slot)
Sent when a feed exports slot is closed.
This signal supports asynchronous handlers.
* **Parameters:**
**slot** (*scrapy.extensions.feedexport.FeedSlot*) – the slot closed
##### feed_exporter_closed
#### scrapy.signals.feed_exporter_closed()
Sent when the feed exports extension is closed,
during the handling of the `spider_closed` signal by the extension,
after all feed exporting has been handled.
This signal supports asynchronous handlers.
##### memusage_warning_reached
#### scrapy.signals.memusage_warning_reached()
Sent by the `MemoryUsage` extension when the
memory usage reaches the warning threshold (`MEMUSAGE_WARNING_MB`).
This signal does not support asynchronous handlers.
#### Request signals
##### request_scheduled
#### scrapy.signals.request_scheduled(request, spider)
Sent when the engine is asked to schedule a `Request`, to be
downloaded later, before the request reaches the scheduler.
Raise `IgnoreRequest` to drop a request before it
reaches the scheduler.
This signal does not support asynchronous handlers.
> [!WARNING]
> Added in version 2.11.2: Allow dropping requests with `IgnoreRequest`.
* **Parameters:**
* **request** (`Request` object) – the request that reached the scheduler
* **spider** (`Spider` object) – the spider that yielded the request
##### request_dropped
#### scrapy.signals.request_dropped(request, spider)
Sent when a `Request`, scheduled by the engine to be
downloaded later, is rejected by the scheduler.
This signal does not support asynchronous handlers.
* **Parameters:**
* **request** (`Request` object) – the request that reached the scheduler
* **spider** (`Spider` object) – the spider that yielded the request
##### request_reached_downloader
#### scrapy.signals.request_reached_downloader(request, spider)
Sent when a `Request` reached downloader.
This signal does not support asynchronous handlers.
* **Parameters:**
* **request** (`Request` object) – the request that reached downloader
* **spider** (`Spider` object) – the spider that yielded the request
##### request_left_downloader
#### scrapy.signals.request_left_downloader(request, spider)
Sent when a `Request` leaves the downloader, even in case of
failure.
This signal does not support asynchronous handlers.
* **Parameters:**
* **request** (`Request` object) – the request that reached the downloader
* **spider** (`Spider` object) – the spider that yielded the request
##### bytes_received
#### scrapy.signals.bytes_received(data, request, spider)
Sent by some download handlers when a group of bytes is
received for a specific request. This signal might be fired multiple
times for the same request, with partial data each time. For instance,
a possible scenario for a 25 kb response would be two signals fired
with 10 kb of data, and a final one with 5 kb of data.
Handlers for this signal can stop the download of a response while it
is in progress by raising the `StopDownload`
exception. Please refer to the topics-stop-response-download topic
for additional information and examples.
This signal does not support asynchronous handlers.
* **Parameters:**
* **data** (`bytes` object) – the data received by the download handler
* **request** (`Request` object) – the request that generated the download
* **spider** (`Spider` object) – the spider associated with the response
##### headers_received
#### scrapy.signals.headers_received(headers, body_length, request, spider)
Sent by some download handlers when the response headers are
available for a given request, before downloading any additional content.
Handlers for this signal can stop the download of a response while it
is in progress by raising the `StopDownload`
exception. Please refer to the topics-stop-response-download topic
for additional information and examples.
This signal does not support asynchronous handlers.
* **Parameters:**
* **headers** (`scrapy.http.headers.Headers` object) – the headers received by the download handler
* **body_length** (int) – expected size of the response body, in bytes
* **request** (`Request` object) – the request that generated the download
* **spider** (`Spider` object) – the spider associated with the response
##### robots_parsed
#### scrapy.signals.robots_parsed(robotparser, request)
> [!WARNING]
> Added in version 2.18.0.
Sent by
`RobotsTxtMiddleware` after it
downloads and parses a `robots.txt` file, for the host that *request*
targets.
This signal supports asynchronous handlers.
* **Parameters:**
* **robotparser** (`RobotParser` object) – the parser holding the parsed `robots.txt` contents
* **request** (`Request` object) – the request that triggered the `robots.txt` download
#### Response signals
##### response_received
#### scrapy.signals.response_received(response, request, spider)
Sent when the engine receives a new `Response` from the
downloader.
This signal does not support asynchronous handlers.
* **Parameters:**
* **response** (`Response` object) – the response received
* **request** (`Request` object) – the request that generated the response
* **spider** (`Spider` object) – the spider for which the response is intended
> [!NOTE]
> The `request` argument might not contain the original request that
> reached the downloader, if a topics-downloader-middleware modifies
> the `Response` object and sets a specific `request`
> attribute.
##### response_downloaded
#### scrapy.signals.response_downloaded(response, request, spider)
Sent by the downloader right after a `Response` is downloaded.
This signal does not support asynchronous handlers.
* **Parameters:**
* **response** (`Response` object) – the response downloaded
* **request** (`Request` object) – the request that generated the response
* **spider** (`Spider` object) – the spider for which the response is intended
Source: /topics/scheduler.md
## Scheduler
The scheduler component receives requests from the engine
and stores them into persistent and/or non-persistent data structures.
It also gets those requests and feeds them back to the engine when it
asks for a next request to be downloaded.
### Overriding the default scheduler
You can use your own custom scheduler class by supplying its full
Python path in the `SCHEDULER` setting.
### Minimal scheduler interface
#### *class* scrapy.core.scheduler.BaseScheduler
The scheduler component is responsible for storing requests received
from the engine, and feeding them back upon request (also to the engine).
The original sources of said requests are:
* Spider: `start` method, requests created for URLs in the `start_urls` attribute, request callbacks
* Spider middleware: `process_spider_output` and `process_spider_exception` methods
* Downloader middleware: `process_request`, `process_response` and `process_exception` methods
The order in which the scheduler returns its stored requests (via the `next_request` method)
plays a great part in determining the order in which those requests are downloaded. See request-order.
The methods defined in this class constitute the minimal interface that the Scrapy engine will interact with.
##### *classmethod* from_crawler(crawler: Crawler) → Self
Factory method which receives the current `Crawler` object as argument.
##### open(spider: Spider) → Deferred[None] | None
Called when the spider is opened by the engine. It receives the spider
instance as argument and it’s useful to execute initialization code.
* **Parameters:**
**spider** (`Spider`) – the spider object for the current crawl
##### close(reason: str) → Deferred[None] | None
Called when the spider is closed by the engine. It receives the reason why the crawl
finished as argument and it’s useful to execute cleaning code.
* **Parameters:**
**reason** (`str`) – a string which describes the reason why the spider was closed
##### *abstractmethod* has_pending_requests() → bool
`True` if the scheduler has enqueued requests, `False` otherwise
##### *abstractmethod* enqueue_request(request: Request) → bool
Process a request received by the engine.
Return `True` if the request is stored correctly, `False` otherwise.
If `False`, the engine will fire a `request_dropped` signal, and
will not make further attempts to schedule the request at a later time.
For reference, the default Scrapy scheduler returns `False` when the
request is rejected by the dupefilter.
##### *abstractmethod* next_request() → Request | None
Return the next `Request` to be processed, or `None`
to indicate that there are no requests to be considered ready at the moment.
Returning `None` implies that no request from the scheduler will be sent
to the downloader in the current reactor cycle. The engine will continue
calling `next_request` until `has_pending_requests` is `False`.
### Default scheduler
#### *class* scrapy.core.scheduler.Scheduler
Default scheduler.
Requests are stored into priority queues
(`SCHEDULER_PRIORITY_QUEUE`) that sort requests by
`priority`.
By default, memory-based priority queues are used for all requests.
When using `JOBDIR`, disk-based priority queues are also created,
and only unserializable requests are stored in the memory-based priority
queues. For a given priority value, requests in memory take precedence over
requests in disk.
Each priority queue stores requests in separate internal queues, one per
priority value. The memory priority queue uses
`SCHEDULER_MEMORY_QUEUE` queues, while the disk priority queue
uses `SCHEDULER_DISK_QUEUE` queues. The internal queues determine
request order when requests have the same priority.
Start requests are stored into separate internal
queues by default, and ordered differently.
Duplicate requests are filtered out with an instance of
`DUPEFILTER_CLASS`.
#### Request order
With default settings, pending requests are stored in a [LIFO](https://en.wikipedia.org/wiki/Stack_(abstract_data_type)) queue
(except for start requests). As a result,
crawling happens in [DFO order](https://en.wikipedia.org/wiki/Depth-first_search), which is usually the most convenient
crawl order. However, you can enforce BFO or a custom
order (except for the first few requests).
##### Start request order
Start requests are sent in the order they are
yielded from `start()`, and given the same
`priority`, other requests take precedence over
start requests.
You can set `SCHEDULER_START_MEMORY_QUEUE` and
`SCHEDULER_START_DISK_QUEUE` to `None` to handle start requests
the same as other requests when it comes to order and priority.
##### Crawling in BFO order
If you do want to crawl in [BFO order](https://en.wikipedia.org/wiki/Breadth-first_search), you can do it by setting the
following settings:
`DEPTH_PRIORITY` = `1`
<br/>
`SCHEDULER_DISK_QUEUE` =
`"scrapy.squeues.PickleFifoDiskQueue"`
<br/>
`SCHEDULER_MEMORY_QUEUE` = `"scrapy.squeues.FifoMemoryQueue"`
<br/>
##### Crawling in a custom order
You can manually set `priority` on requests to
force a specific request order.
##### Concurrency affects order
While pending requests are below the configured values of
`CONCURRENT_REQUESTS` or
`CONCURRENT_REQUESTS_PER_DOMAIN`, those requests are sent
concurrently.
As a result, the first few requests of a crawl may not follow the desired
order. Lowering those settings to `1` enforces the desired order except
for the very first request, but it significantly slows down the crawl as a
whole.
#### Job directory contents
> [!WARNING]
> The files that this class generates in the job directory are an implementation detail, and may change without a
> warning in a future version of Scrapy. Do not rely on the following
> information for anything other than debugging purposes.
When using `JOBDIR`, this scheduler class:
- Creates a directory named `requests.queue` inside the job
directory, meant to keep track of all requests stored in
the scheduler (i.e. not downloaded yet).
- Generates inside that directory an `active.json` file with a JSON
representation of the state (`startprios`) of
`SCHEDULER_PRIORITY_QUEUE`.
The file is generated whenever the job stops (cleanly) and is loaded
when resuming the job.
- Instantiates the configured `SCHEDULER_PRIORITY_QUEUE` with
`requests.queue/` as persistence directory (*key*) and
`SCHEDULER_DISK_QUEUE` as *downstream_queue_cls*. The priority
queue may create additional files and directories inside that
directory, directly or though instances of
`SCHEDULER_DISK_QUEUE`.
This scheduler class also uses the configured `DUPEFILTER_CLASS`,
which may also write data inside the job directory.
##### *classmethod* from_crawler(crawler: Crawler) → Self
Factory method which receives the current `Crawler` object as argument.
##### \_\_init_\_(dupefilter: BaseDupeFilter, jobdir: str | None = None, dqclass: type[BaseQueue] | None = None, mqclass: type[BaseQueue] | None = None, logunser: bool = False, stats: StatsCollector | None = None, pqclass: type[ScrapyPriorityQueue] | None = None, crawler: Crawler | None = None)
Initialize the scheduler.
* **Parameters:**
* **dupefilter** (`scrapy.dupefilters.BaseDupeFilter` instance or similar:
any class that implements the BaseDupeFilter interface) – An object responsible for checking and filtering duplicate requests.
The value for the `DUPEFILTER_CLASS` setting is used by default.
* **jobdir** (`str` or `None`) – The path of a directory to be used for persisting the crawl’s state.
The value for the `JOBDIR` setting is used by default.
See topics-jobs.
* **dqclass** (*type*) – A class to be used as persistent request queue.
The value for the `SCHEDULER_DISK_QUEUE` setting is used by default.
* **mqclass** (*type*) – A class to be used as non-persistent request queue.
The value for the `SCHEDULER_MEMORY_QUEUE` setting is used by default.
* **logunser** (*bool*) – A boolean that indicates whether or not unserializable requests should be logged.
The value for the `SCHEDULER_DEBUG` setting is used by default.
* **stats** (`scrapy.statscollectors.StatsCollector` instance or similar:
any class that implements the StatsCollector interface) – A stats collector object to record stats about the request scheduling process.
The value for the `STATS_CLASS` setting is used by default.
* **pqclass** (*type*) – A class to be used as priority queue for requests.
The value for the `SCHEDULER_PRIORITY_QUEUE` setting is used by default.
* **crawler** (`scrapy.crawler.Crawler`) – The crawler object corresponding to the current crawl.
##### has_pending_requests() → bool
`True` if the scheduler has enqueued requests, `False` otherwise
##### open(spider: Spider) → Deferred[None] | None
1. initialize the memory queue
2. initialize the disk queue if the `jobdir` argument wasn’t empty
3. return the result of the dupefilter’s `open` method
##### close(reason: str) → Deferred[None] | None
1. dump pending requests to disk if there is a disk queue
2. return the result of the dupefilter’s `close` method
##### enqueue_request(request: Request) → bool
Unless the received request is filtered out by the Dupefilter, attempt to push
it into the disk queue, falling back to pushing it into the memory queue.
Increment the appropriate stats, such as: `scheduler/enqueued`,
`scheduler/enqueued/disk`, `scheduler/enqueued/memory`.
Return `True` if the request was stored successfully, `False` otherwise.
##### next_request() → Request | None
Return a `Request` object from the memory queue,
falling back to the disk queue if the memory queue is empty.
Return `None` if there are no more enqueued requests.
Increment the appropriate stats, such as: `scheduler/dequeued`,
`scheduler/dequeued/disk`, `scheduler/dequeued/memory`.
##### \_\_len_\_() → int
Return the total amount of enqueued requests
### Priority queues
#### *class* scrapy.pqueues.DownloaderAwarePriorityQueue(crawler: Crawler, downstream_queue_cls: type[QueueProtocol], key: str, slot_startprios: dict[str, Iterable[int]] | None = None, \*, start_queue_cls: type[QueueProtocol] | None = None)
PriorityQueue which takes Downloader activity into account:
domains (slots) with the least amount of active downloads are dequeued
first.
#### Disk persistence
> [!WARNING]
> The files that this class generates on disk are an
> implementation detail, and may change without a warning in a future
> version of Scrapy. Do not rely on the following information for
> anything other than debugging purposes.
When a component instantiates this class with a non-empty *key* argument,
*key* is used as a persistence directory, and inside that directory this
class creates a subdirectory per download slot (domain).
Those subdirectories are named after the corresponding download slot, with
path-unsafe characters replaced by underscores and an MD5 hash suffix to
avoid collisions.
For each download slot, this class creates an instance of
`ScrapyPriorityQueue` with the download slot subdirectory as *key*
and its own *downstream_queue_cls*.
#### *class* scrapy.pqueues.ScrapyPriorityQueue(crawler: Crawler, downstream_queue_cls: type[QueueProtocol], key: str, startprios: Iterable[int] = (), \*, start_queue_cls: type[QueueProtocol] | None = None)
A priority queue implemented using multiple internal queues (typically,
FIFO queues). It uses one internal queue for each priority value. The
internal queue must implement the following methods:
> * push(obj)
> * pop()
> * close()
> * \_\_len_\_()
Optionally, the queue could provide a `peek` method, that should return
the next object to be returned by `pop`, but without removing it from the
queue.
`__init__` method of ScrapyPriorityQueue receives a downstream_queue_cls
argument, which is a class used to instantiate a new (internal) queue when
a new priority is allocated.
Only integer priorities should be used. Lower numbers are higher
priorities.
startprios is a sequence of priorities to start with. If the queue was
previously closed leaving some priority buckets non-empty, those priorities
should be passed in startprios.
#### Disk persistence
> [!WARNING]
> The files that this class generates on disk are an
> implementation detail, and may change without a warning in a future
> version of Scrapy. Do not rely on the following information for
> anything other than debugging purposes.
When a component instantiates this class with a non-empty *key* argument,
*key* is used as a persistence directory.
For every request enqueued, this class checks:
- Whether the request is a start request or not.
- The `priority` of the request.
For each combination of the above seen, this class creates an instance of
*downstream_queue_cls* (or *start_queue_cls* for start requests if it was
passed) with *key* set to a subdirectory of the persistence directory,
named as the negated request priority (e.g. `-1`), with an `s` suffix
in case of a start request (e.g. `-1s`).
Source: /topics/exporters.md
## Item Exporters
Once you have scraped your items, you often want to persist or export those
items, to use the data in some other application. That is, after all, the whole
purpose of the scraping process.
For this purpose Scrapy provides a collection of Item Exporters for different
output formats, such as XML, CSV or JSON.
### Using Item Exporters
If you are in a hurry, and just want to use an Item Exporter to output scraped
data see the topics-feed-exports. Otherwise, if you want to know how
Item Exporters work or need more custom functionality (not covered by the
default exports), continue reading below.
In order to use an Item Exporter, you must instantiate it with its required
args. Each Item Exporter requires different arguments, so check each exporter
documentation to be sure, in topics-exporters-reference. After you have
instantiated your exporter, you have to:
1. call the method `start_exporting()` in order to
signal the beginning of the exporting process
2. call the `export_item()` method for each item you want
to export
3. and finally call the `finish_exporting()` to signal
the end of the exporting process
Here you can see an Item Pipeline which uses multiple
Item Exporters to group scraped items to different files according to the
value of one of their fields:
```python
from itemadapter import ItemAdapter
from scrapy.exporters import XmlItemExporter
class PerYearXmlExportPipeline:
"""Distribute items across multiple XML files according to their 'year' field"""
def open_spider(self, spider):
self.year_to_exporter = {}
def close_spider(self, spider):
for exporter, xml_file in self.year_to_exporter.values():
exporter.finish_exporting()
xml_file.close()
def _exporter_for_item(self, item):
adapter = ItemAdapter(item)
year = adapter["year"]
if year not in self.year_to_exporter:
xml_file = open(f"{year}.xml", "wb")
exporter = XmlItemExporter(xml_file)
exporter.start_exporting()
self.year_to_exporter[year] = (exporter, xml_file)
return self.year_to_exporter[year][0]
def process_item(self, item):
exporter = self._exporter_for_item(item)
exporter.export_item(item)
return item
```
### Serialization of item fields
By default, the field values are passed unmodified to the underlying
serialization library, and the decision of how to serialize them is delegated
to each particular serialization library.
However, you can customize how each field value is serialized *before it is
passed to the serialization library*.
There are two ways to customize how a field will be serialized, which are
described next.
#### 1. Declaring a serializer in the field
Every item type except `dict` lets you declare a
serializer in the field metadata. The serializer
must be a callable which receives a value and returns its serialized form.
Example:
```python
from dataclasses import dataclass, field
def serialize_price(value):
return f"$ {str(value)}"
@dataclass
class Product:
name: str
price: float = field(metadata={"serializer": serialize_price})
```
#### 2. Overriding the serialize_field() method
You can also override the `serialize_field()` method to
customize how your field value will be exported.
Make sure you call the base class `serialize_field()` method
after your custom code.
Example:
```python
from scrapy.exporters import XmlItemExporter
class ProductXmlExporter(XmlItemExporter):
def serialize_field(self, field, name, value):
if name == "price":
return f"$ {str(value)}"
return super().serialize_field(field, name, value)
```
### Writing your own item exporter
To write an item exporter, subclass `BaseItemExporter` and implement
`export_item()`, where
`get_serialized_fields()` gives you the `(name, value)`
pairs to export.
To make your exporter available to the feed exports, list it in the `FEED_EXPORTERS` setting. Feed
exports build it with the output file as the first
positional argument, and with the `fields`, `encoding` and `indent`
feed options and every key of `item_export_kwargs` as
keyword arguments, so your `__init__` method must forward unknown keyword
arguments to `BaseItemExporter`.
The file object belongs to whoever opened it, i.e. to the feed storage in the
case of feed exports, which also closes it. If you need a text file, for
example to use `csv.writer()` or another Python API that does not accept a
binary file, wrap it with `io.TextIOWrapper` and call
`detach()` on the wrapper in
`finish_exporting()`; otherwise the wrapper closes the
underlying file when it is garbage-collected.
For example, the following item exporter writes items as blocks of
`name: value` lines:
```python
from io import TextIOWrapper
from scrapy.exporters import BaseItemExporter
class TextItemExporter(BaseItemExporter):
def __init__(self, file, item_separator="\n", **kwargs):
super().__init__(**kwargs)
self.item_separator = item_separator
self.stream = TextIOWrapper(
file, encoding=self.encoding or "utf-8", write_through=True
)
def export_item(self, item):
for name, value in self.get_serialized_fields(item):
print(f"{name}: {value}", file=self.stream)
self.stream.write(self.item_separator)
def finish_exporting(self):
self.stream.detach()
```
To use it as the `txt` feed format:
```python
FEED_EXPORTERS = {"txt": "myproject.exporters.TextItemExporter"}
FEEDS = {
"items.txt": {
"format": "txt",
"item_export_kwargs": {"item_separator": "---\n"},
},
}
```
### Built-in Item Exporters reference
Here is a list of the Item Exporters bundled with Scrapy. Some of them contain
output examples, which assume you’re exporting these two items:
```python
Item(name="Color TV", price="1200")
Item(name="DVD player", price="200")
```
#### BaseItemExporter
#### *class* scrapy.exporters.BaseItemExporter(fields_to_export=None, export_empty_fields=False, encoding=None, indent=None, dont_fail=False)
This is the (abstract) base class for all Item Exporters. It provides
support for common features used by all (concrete) Item Exporters, such as
defining what fields to export, whether to export empty fields, or which
encoding to use.
These features can be configured through the `__init__` method arguments which
populate their respective instance attributes: `fields_to_export`,
`export_empty_fields`, `encoding`, `indent`.
##### export_item(item)
Exports the given item. This method must be implemented in subclasses.
##### get_serialized_fields(item: Any, default_value: Any = None, include_empty: bool | None = None) → Iterable[tuple[str, Any]]
Return the fields of *item* to export, as an iterable of
`(name, serialized_value)` tuples, taking `fields_to_export`
into account and applying `serialize_field()` to every value.
Fields missing from *item* are exported with *default_value*.
*include_empty* overrides `export_empty_fields`.
##### serialize_field(field, name, value)
Return the serialized value for the given field. You can override this
method (in your custom Item Exporters) if you want to control how a
particular field or value will be serialized/exported.
By default, this method looks for a serializer declared in the item
field and returns the result of applying
that serializer to the value. If no serializer is found, it returns the
value unchanged.
* **Parameters:**
* **field** (`Field` object or a `dict` instance) – the field being serialized. If the source item object does not define field metadata, *field* is an empty
`dict`.
* **name** (*str*) – the name of the field being serialized
* **value** – the value being serialized
##### start_exporting()
Signal the beginning of the exporting process. Some exporters may use
this to generate some required header (for example, the
`XmlItemExporter`). You must call this method before exporting any
items.
##### finish_exporting()
Signal the end of the exporting process. Some exporters may use this to
generate some required footer (for example, the
`XmlItemExporter`). You must always call this method after you
have no more items to export.
##### fields_to_export
Fields to export, their order [^1] and their output names.
Possible values are:
- `None` (all fields [^2], default)
Fields are exported in declaration order, i.e. the order in which
they are defined in the item class. For
`dict` items, which have no declared fields, the key order of
each item is used instead.
> [!WARNING]
> Changed in version 2.18.0: Fields of non-`dict` items used to be exported in the
> order in which they had been populated, except in
> `CsvItemExporter`, which has always used declaration order.
- A list of fields:
```python
["field1", "field2"]
```
- A dict where keys are fields and values are output names:
```python
{"field1": "Field 1", "field2": "Field 2"}
```
[^1]: Not all exporters respect the specified field order.
[^2]: When using item objects that do not expose all their possible fields, exporters that do not support exporting a different subset of fields per item will only export the fields found in the first item exported.
##### export_empty_fields
Whether to include empty/unpopulated item fields in the exported data.
Defaults to `False`. Some exporters (like `CsvItemExporter`)
ignore this attribute and always export all empty fields.
This option is ignored for dict items.
##### encoding
The output character encoding.
##### indent
Amount of spaces used to indent the output on each level. Defaults to `None`.
* `indent=None` selects the most compact representation,
all items in the same line with no indentation
* `indent<=0` each item on its own line, no indentation
* `indent>0` each item on its own line, indented with the provided numeric value
#### PythonItemExporter
#### *class* scrapy.exporters.PythonItemExporter(\*, dont_fail: bool = False, \*\*kwargs: Any)
This is a base class for item exporters that extends
`BaseItemExporter` with support for nested items.
It serializes items to built-in Python types, so that any serialization
library (e.g. `json` or [msgpack](https://pypi.org/project/msgpack/)) can be used on top of it.
#### XmlItemExporter
#### *class* scrapy.exporters.XmlItemExporter(file, item_element='item', root_element='items', \*\*kwargs)
Exports items in XML format to the specified file object.
* **Parameters:**
* **file** – the file-like object to use for exporting the data. Its `write` method should
accept `bytes` (a disk file opened in binary mode, a `io.BytesIO` object, etc)
* **root_element** (*str*) – The name of root element in the exported XML.
* **item_element** (*str*) – The name of each item element in the exported XML.
The additional keyword arguments of this `__init__` method are passed to the
`BaseItemExporter` `__init__` method.
A typical output of this exporter would be:
```xml
<?xml version="1.0" encoding="utf-8"?>
<items>
<item>
<name>Color TV</name>
<price>1200</price>
</item>
<item>
<name>DVD player</name>
<price>200</price>
</item>
</items>
```
Unless overridden in the `serialize_field()` method, multi-valued fields are
exported by serializing each value inside a `<value>` element. This is for
convenience, as multi-valued fields are very common.
For example, the item:
```python
Item(name=["John", "Doe"], age="23")
```
Would be serialized as:
```xml
<?xml version="1.0" encoding="utf-8"?>
<items>
<item>
<name>
<value>John</value>
<value>Doe</value>
</name>
<age>23</age>
</item>
</items>
```
#### CsvItemExporter
#### *class* scrapy.exporters.CsvItemExporter(file, include_headers_line=True, join_multivalued=',', errors=None, \*\*kwargs)
Exports items in CSV format to the given file-like object. If the
`fields_to_export` attribute is set, it will be used to define the
CSV columns, their order and their column names. The
`export_empty_fields` attribute has no effect on this exporter.
* **Parameters:**
* **file** – the file-like object to use for exporting the data. Its `write` method should
accept `bytes` (a disk file opened in binary mode, a `io.BytesIO` object, etc)
* **include_headers_line** (*bool*) – If enabled, makes the exporter output a header
line with the field names taken from
`BaseItemExporter.fields_to_export` or the first exported item fields.
* **join_multivalued** (*str*) – The char (or chars) that will be used for joining
multi-valued fields, if found.
* **errors** (*str*) – The optional string that specifies how encoding and decoding
errors are to be handled. For more information see
`io.TextIOWrapper`.
The additional keyword arguments of this `__init__` method are passed to the
`BaseItemExporter` `__init__` method, and the leftover arguments to the
`csv.writer()` function, so you can use any `csv.writer()` function
argument to customize this exporter.
A typical output of this exporter would be:
```
name,price
Color TV,1200
DVD player,200
```
#### PickleItemExporter
#### *class* scrapy.exporters.PickleItemExporter(file, protocol=4, \*\*kwargs)
Exports items in pickle format to the given file-like object.
* **Parameters:**
* **file** – the file-like object to use for exporting the data. Its `write` method should
accept `bytes` (a disk file opened in binary mode, a `io.BytesIO` object, etc)
* **protocol** (*int*) – The pickle protocol to use.
For more information, see `pickle`.
The additional keyword arguments of this `__init__` method are passed to the
`BaseItemExporter` `__init__` method.
Pickle isn’t a human readable format, so no output examples are provided.
#### PprintItemExporter
#### *class* scrapy.exporters.PprintItemExporter(file, \*\*kwargs)
Exports items in pretty print format to the specified file object.
* **Parameters:**
**file** – the file-like object to use for exporting the data. Its `write` method should
accept `bytes` (a disk file opened in binary mode, a `io.BytesIO` object, etc)
The additional keyword arguments of this `__init__` method are passed to the
`BaseItemExporter` `__init__` method.
A typical output of this exporter would be:
```python
{"name": "Color TV", "price": "1200"}
{"name": "DVD player", "price": "200"}
```
Longer lines (when present) are pretty-formatted.
#### JsonItemExporter
#### *class* scrapy.exporters.JsonItemExporter(file, \*\*kwargs)
Exports items in JSON format to the specified file-like object, writing all
objects as a list of objects. The additional `__init__` method arguments are
passed to the `BaseItemExporter` `__init__` method, and the leftover
arguments to the `JSONEncoder` `__init__` method, so you can use any
`JSONEncoder` `__init__` method argument to customize this exporter.
* **Parameters:**
**file** – the file-like object to use for exporting the data. Its `write` method should
accept `bytes` (a disk file opened in binary mode, a `io.BytesIO` object, etc)
A typical output of this exporter would be:
```json
[{"name": "Color TV", "price": "1200"},
{"name": "DVD player", "price": "200"}]
```
> [!WARNING]
> JSON is very simple and flexible serialization format, but it
> doesn’t scale well for large amounts of data since incremental (aka.
> stream-mode) parsing is not well supported (if at all) among JSON parsers
> (on any language), and most of them just parse the entire object in
> memory. If you want the power and simplicity of JSON with a more
> stream-friendly format, consider using `JsonLinesItemExporter`
> instead, or splitting the output in multiple chunks.
#### JsonLinesItemExporter
#### *class* scrapy.exporters.JsonLinesItemExporter(file, \*\*kwargs)
Exports items in JSON format to the specified file-like object, writing one
JSON-encoded item per line. The additional `__init__` method arguments are passed
to the `BaseItemExporter` `__init__` method, and the leftover arguments to
the `JSONEncoder` `__init__` method, so you can use any
`JSONEncoder` `__init__` method argument to customize this exporter.
* **Parameters:**
**file** – the file-like object to use for exporting the data. Its `write` method should
accept `bytes` (a disk file opened in binary mode, a `io.BytesIO` object, etc)
A typical output of this exporter would be:
```json
{"name": "Color TV", "price": "1200"}
{"name": "DVD player", "price": "200"}
```
Unlike the one produced by `JsonItemExporter`, the format produced by
this exporter is well suited for serializing large amounts of data.
#### MarshalItemExporter
#### *class* scrapy.exporters.MarshalItemExporter(file: BytesIO, \*\*kwargs: Any)
Exports items in a Python-specific binary format (see
`marshal`).
* **Parameters:**
**file** – The file-like object to use for exporting the data. Its
`write` method should accept `bytes` (a disk file
opened in binary mode, a `BytesIO` object, etc)
Source: /topics/download-handlers.md
## Download handlers
Download handlers are Scrapy components used to
download requests and produce responses from
them.
### Using download handlers
The `DOWNLOAD_HANDLERS_BASE` and `DOWNLOAD_HANDLERS` settings
tell Scrapy which handler is responsible for a given URL scheme. Their values
are merged into a mapping from scheme names to handler classes. When Scrapy
initializes it creates instances of all configured download handlers (except
for lazy ones) and stores them in a similar
mapping. When Scrapy needs to download a request it extracts the scheme from
its URL, finds the handler for this scheme, passes the request to it and gets a
response from it. If there is no handler for the scheme, the request is not
downloaded and a `NotSupported` exception is raised.
The `DOWNLOAD_HANDLERS_BASE` setting contains the default mapping of
handlers. You can use the `DOWNLOAD_HANDLERS` setting to add handlers
for additional schemes and to replace or disable default ones:
```python
DOWNLOAD_HANDLERS = {
# disable support for ftp:// requests
"ftp": None,
# replace the default one for http://
"http": "my.download_handlers.HttpHandler",
# http:// and https:// are different schemes,
# even though they may use the same handler
"https": "my.download_handlers.HttpHandler",
# support for any custom scheme can be added
"sftp": "my.download_handlers.SftpHandler",
}
```
> [!NOTE]
> ### See also
>
> security-unencrypted-protocols and
> security-local-resources, for the security implications of the
> default `http`, `ftp`, `file` and `data` handlers.
#### Replacing HTTP(S) download handlers
While Scrapy provides a default handler for `http` and `https` schemes,
users may want to use a different handler, provided by Scrapy or by some
3rd-party package. There are several considerations to keep in mind related to
this.
First of all, as `http` and `https` are separate schemes, they need
separate entries in the `DOWNLOAD_HANDLERS` setting, even though it’s
likely that the same handler class will be used for both schemes.
Additionally, some of the Scrapy settings, like `DOWNLOAD_MAXSIZE`,
are honored by the default HTTP(S) handler but not necessarily by alternative
ones. The same may apply to other Scrapy features, e.g. the
`bytes_received` and `headers_received` signals.
#### Lazy instantiation of download handlers
A download handler can be marked as “lazy” by setting its `lazy` class
attribute to `True`. Such handlers are only instantiated when they need to
download their first request. This may be useful when the instantiation is slow
or requires dependencies that are not always available, and the handler is not
needed on every spider run. For example, `the built-in S3 handler` is lazy.
### Writing your own download handler
A download handler is a component that defines
the following API:
#### *class* scrapy.core.downloader.handlers.DownloadHandlerProtocol(\*args, \*\*kwargs)
Interface that download handlers must
implement.
Besides implementing this protocol, the contract of a download handler
includes **never** calling `crawler.engine.download_async()`.
##### lazy *: bool*
Whether to delay instantiation of the handler; see lazy.
##### *async* download_request(request: Request) → Response
Download *request* and return a response.
##### *async* close() → None
Clean up any resources used by the handler.
An optional base class for custom handlers is provided:
#### *class* scrapy.core.downloader.handlers.base.BaseDownloadHandler(crawler: Crawler)
Optional base class for download handlers.
##### *classmethod* from_crawler(crawler: Crawler) → Self
### Exceptions raised by download handlers
> [!WARNING]
> Added in version 2.15.0.
The built-in download handlers raise Scrapy-specific exceptions instead of
implementation-specific ones, so that code that handles these exceptions can be
written in a generic way. We recommend custom download handlers to also use
these exceptions.
#### *exception* scrapy.exceptions.CannotResolveHostError
Indicates that the provided hostname cannot be resolved.
#### *exception* scrapy.exceptions.DownloadCancelledError
Indicates that a request download was cancelled.
#### *exception* scrapy.exceptions.DownloadConnectionRefusedError
Indicates that a connection was refused by the server.
#### *exception* scrapy.exceptions.DownloadFailedError
Indicates that a request download has failed.
#### *exception* scrapy.exceptions.DownloadTimeoutError
Indicates that a request download has timed out.
#### *exception* scrapy.exceptions.ResponseDataLossError
Indicates that Scrapy couldn’t get a complete response.
#### *exception* scrapy.exceptions.UnsupportedURLSchemeError
Indicates that the URL scheme is not supported.
### Built-in HTTP download handlers reference
Scrapy ships several handlers for HTTP and HTTPS requests. While all of them
support basic features, they may differ in support of specific Scrapy features
and settings and HTTP protocol features. See the documentation of specific
handlers and specific settings for more information. Additionally, as the
underlying HTTP client implementations differ between handlers, the behavior of
specific websites may be different when doing the same Scrapy requests but
using different handlers.
Here is a comparison of some features of the built-in HTTP handlers, see the
individual handler docs for more differences:
| Handler | Requirements | HTTP | Proxies | Bad headers | TLS |
|-----------|------------------------------|--------|-------------|----------------|----------------|
| `Aiohttp` | asyncio | 1.1 | HTTP | Fail | Stdlib `ssl` |
| `H2` | Reactor, twisted-http2 extra | 2 | None | Not applicable | `cryptography` |
| `HTTP11` | Reactor | 1.1 | HTTP | Skip bad | `cryptography` |
| `Httpx` | asyncio, httpx extra | 1.1, 2 | HTTP, SOCKS | Fail | Stdlib `ssl` |
Bad header handling is what a handler does when a response has a bad header
line, e.g. one with no colon in it, which some servers send. Handlers that skip
bad header lines, like web browsers do, still parse the header lines that follow
them; other handlers also lose those, or cannot download such responses at all.
You can find additional HTTP download handlers in the
[scrapy-download-handlers-incubator](https://github.com/scrapy-plugins/scrapy-download-handlers-incubator) package. This package is made by the Scrapy
developers and contains experimental handlers that may be included in some
later Scrapy version but can already be used. Please refer to the documentation
of this package for more information.
#### AiohttpDownloadHandler
> [!WARNING]
> Added in version 2.19.0.
#### *class* scrapy.core.downloader.handlers._aiohttp.AiohttpDownloadHandler(crawler: Crawler)
Supported schemes: `http`, `https`.
<br/>
Lazy: no.
<br/>
Requires asyncio support: yes.
<br/>
Requires a Twisted reactor: no.
<br/>
This handler supports `http://host/path` and `https://host/path` URLs and
uses the HTTP/1.1 protocol for them.
It’s implemented using the [aiohttp](https://docs.aiohttp.org/) library.
If you want to use this handler you need to replace the default ones for the
`http` and `https` schemes:
```python
DOWNLOAD_HANDLERS = {
"http": "scrapy.core.downloader.handlers._aiohttp.AiohttpDownloadHandler",
"https": "scrapy.core.downloader.handlers._aiohttp.AiohttpDownloadHandler",
}
```
##### Features and limitations
> [!WARNING]
> This handler is experimental, and not yet recommended for production
> environments. Future Scrapy versions may introduce related changes without
> a deprecation period or warning or even remove it altogether.
| HTTP proxies | Yes |
|---------------------------|-------------------------------------|
| SOCKS proxies | No (not supported by the library) |
| HTTP/2 | No (not supported by the library) |
| Bad header handling | Fail (not supported by the library) |
| `response.certificate` | DER bytes |
| Per-request `bindaddress` | No (not supported by the library) |
| TLS implementation | Standard library `ssl` |
Other limitations:
- HTTPS proxies for HTTPS destinations are not supported on Python < 3.11.
#### H2DownloadHandler
> [!NOTE]
> Requires the twisted-http2 extra.
#### *class* scrapy.core.downloader.handlers.http2.H2DownloadHandler(crawler: Crawler)
Supported scheme: `https`.
<br/>
Lazy: yes.
<br/>
Requires asyncio support: no.
<br/>
Requires a Twisted reactor: yes.
<br/>
This handler supports `https://host/path` URLs and uses the HTTP/2 protocol
for them.
It’s implemented using `twisted.web.client` and the `h2` library.
If you want to use this handler you need to replace the default one for the
`https` scheme:
```python
DOWNLOAD_HANDLERS = {
"https": "scrapy.core.downloader.handlers.http2.H2DownloadHandler",
}
```
##### Features and limitations
| HTTP proxies | No (not implemented) |
|---------------------------|-------------------------------------------|
| SOCKS proxies | No (not supported by the library) |
| HTTP/2 | Yes |
| Bad header handling | Not applicable (HTTP/2 only) |
| `response.certificate` | `twisted.internet.ssl.Certificate` object |
| Per-request `bindaddress` | Yes |
| TLS implementation | `pyOpenSSL`/`cryptography` |
Other limitations:
- No support for HTTP/1.1.
- IPv6 support requires setting `TWISTED_DNS_RESOLVER`
to `scrapy.resolver.CachingHostnameResolver`.
Known limitations of the HTTP/2 support:
- No support for HTTP/2 Cleartext (h2c), since no major browser supports
HTTP/2 unencrypted (refer [http2 faq](https://http2.github.io/faq/#does-http2-require-encryption)).
- No support for [server pushes](https://datatracker.ietf.org/doc/html/rfc7540#section-8.2), which are ignored.
#### HTTP11DownloadHandler
#### *class* scrapy.core.downloader.handlers.http11.HTTP11DownloadHandler(crawler: Crawler)
Supported schemes: `http`, `https`.
<br/>
Lazy: no.
<br/>
Requires asyncio support: no.
<br/>
Requires a Twisted reactor: yes.
<br/>
This handler supports `http://host/path` and `https://host/path` URLs and
uses the HTTP/1.1 protocol for them.
It’s implemented using `twisted.web.client`.
##### Features and limitations
| HTTP proxies | Yes |
|---------------------------|-------------------------------------------|
| SOCKS proxies | No (not supported by the library) |
| HTTP/2 | No (implemented as a separate handler) |
| Bad header handling | Skip bad, like web browsers do |
| `response.certificate` | `twisted.internet.ssl.Certificate` object |
| Per-request `bindaddress` | Yes |
| TLS implementation | `pyOpenSSL`/`cryptography` |
> [!WARNING]
> Changed in version 2.18.0: Bad header lines with no colon in them are now skipped, instead of making
> the whole response impossible to download.
Other limitations:
- IPv6 support requires setting `TWISTED_DNS_RESOLVER`
to `scrapy.resolver.CachingHostnameResolver`.
- HTTPS proxies to HTTPS destinations are not supported.
#### HttpxDownloadHandler
> [!NOTE]
> Requires the httpx extra.
> [!WARNING]
> Added in version 2.15.0.
#### *class* scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler(crawler: Crawler)
Supported schemes: `http`, `https`.
<br/>
Lazy: no.
<br/>
Requires asyncio support: yes.
<br/>
Requires a Twisted reactor: no.
<br/>
This handler supports `http://host/path` and `https://host/path` URLs and
uses the HTTP/1.1 or HTTP/2 protocol for them.
It’s implemented using the [httpx2](https://httpx2.pydantic.dev/) library.
If you want to use this handler you need to replace the default ones for the
`http` and `https` schemes:
```python
DOWNLOAD_HANDLERS = {
"http": "scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler",
"https": "scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler",
}
```
##### Features and limitations
> [!WARNING]
> This handler is experimental, and not yet recommended for production
> environments. Future Scrapy versions may introduce related changes without
> a deprecation period or warning or even remove it altogether.
| HTTP proxies | Yes |
|---------------------------|-------------------------------------|
| SOCKS proxies | Yes (SOCKS5) |
| HTTP/2 | Yes |
| Bad header handling | Fail (not supported by the library) |
| `response.certificate` | DER bytes |
| Per-request `bindaddress` | No (not supported by the library) |
| TLS implementation | Standard library `ssl` |
Other limitations:
- The handler creates a separate connection pool for each proxy URL (due to
limitations of `httpx`) which may lead to higher resource usage when
using proxy rotation.
##### HTTPX_HTTP2_ENABLED
> [!WARNING]
> Added in version 2.17.0.
Default: `False`
Whether to enable HTTP/2 support in this handler.
### Built-in non-HTTP download handlers reference
#### DataURIDownloadHandler
#### *class* scrapy.core.downloader.handlers.datauri.DataURIDownloadHandler(crawler: Crawler)
Supported scheme: `data`.
<br/>
Lazy: no.
<br/>
Requires asyncio support: no.
<br/>
Requires a Twisted reactor: no.
<br/>
This handler supports RFC 2397 `data:content/type;base64,` data URIs.
#### FileDownloadHandler
#### *class* scrapy.core.downloader.handlers.file.FileDownloadHandler(crawler: Crawler)
Supported scheme: `file`.
<br/>
Lazy: no.
<br/>
Requires asyncio support: no.
<br/>
Requires a Twisted reactor: no.
<br/>
This handler supports `file:///path` local file URIs. It doesn’t
support remote files.
#### FTPDownloadHandler
#### *class* scrapy.core.downloader.handlers.ftp.FTPDownloadHandler(crawler: Crawler)
Supported scheme: `ftp`.
<br/>
Lazy: no.
<br/>
Requires asyncio support: no.
<br/>
Requires a Twisted reactor: yes.
<br/>
This handler supports `ftp://host/path` FTP URIs.
It’s implemented using `twisted.protocols.ftp`.
#### S3DownloadHandler
> [!NOTE]
> Requires the s3 extra.
#### *class* scrapy.core.downloader.handlers.s3.S3DownloadHandler(crawler: Crawler)
Supported scheme: `s3`.
<br/>
Lazy: yes.
<br/>
Requires asyncio support: no.
<br/>
Requires a Twisted reactor: no.
<br/>
This handler supports `s3://bucket/path` S3 URIs.
It’s implemented using the [botocore](https://github.com/boto/botocore) library.
Source: /topics/components.md
## Components
A Scrapy component is any class whose objects are built using
`build_from_crawler()`.
That includes the classes that you may assign to the following settings:
- `ADDONS`
- `DOWNLOAD_HANDLERS`
- `DOWNLOADER_MIDDLEWARES`
- `DUPEFILTER_CLASS`
- `EXTENSIONS`
- `FEED_EXPORTERS`
- `FEED_STORAGES`
- `ITEM_PIPELINES`
- `SCHEDULER`
- `SCHEDULER_DISK_QUEUE`
- `SCHEDULER_MEMORY_QUEUE`
- `SCHEDULER_PRIORITY_QUEUE`
- `SCHEDULER_START_DISK_QUEUE`
- `SCHEDULER_START_MEMORY_QUEUE`
- `SPIDER_MIDDLEWARES`
- `TWISTED_DNS_RESOLVER`
Third-party Scrapy components may also let you define additional Scrapy
components, usually configurable through settings, to
modify their behavior.
### Initializing from the crawler
Any Scrapy component may optionally define the following class method:
#### *classmethod* from_crawler(cls, crawler: scrapy.crawler.Crawler, \*args, \*\*kwargs)
Return an instance of the component based on *crawler*.
*args* and *kwargs* are component-specific arguments that some components
receive. However, most components do not get any arguments, and instead
use settings.
If a component class defines this method, this class method is called to
create any instance of the component.
The *crawler* object provides access to all Scrapy core components like
settings and signals,
allowing the component to access them and hook its functionality into
Scrapy.
### Settings
Components can be configured through settings.
Components can read any setting from the
`settings` attribute of the
`Crawler` object they can get for initialization. That includes both built-in and custom settings.
For example:
```python
class MyExtension:
@classmethod
def from_crawler(cls, crawler):
settings = crawler.settings
return cls(settings.getbool("LOG_ENABLED"))
def __init__(self, log_is_enabled=False):
if log_is_enabled:
print("log is enabled!")
```
Components do not need to declare their custom settings programmatically.
However, they should document them, so that users know they exist and how to
use them.
It is a good practice to prefix custom settings with the name of the component,
to avoid collisions with custom settings of other existing (or future)
components. For example, an extension called `WarcCaching` could prefix its
custom settings with `WARC_CACHING_`.
Another good practice, mainly for components meant for component priority
dictionaries, is to provide a boolean setting
called `<PREFIX>_ENABLED` (e.g. `WARC_CACHING_ENABLED`) to allow toggling
that component on and off without changing the component priority dictionary
setting. You can usually check the value of such a setting during
initialization, and if `False`, raise
`NotConfigured`.
When choosing a name for a custom setting, it is also a good idea to have a
look at the names of built-in settings, to try to
maintain consistency with them.
### Enforcing requirements
Sometimes, your components may only be intended to work under certain
conditions. For example, they may require a minimum version of Scrapy to work as
intended, or they may require certain settings to have specific values.
In addition to describing those conditions in the documentation of your
component, it is a good practice to raise an exception from the `__init__`
method of your component if those conditions are not met at run time.
In the case of downloader middlewares,
extensions, item pipelines, and spider middlewares, you should raise
`NotConfigured`, passing a description of the issue as
a parameter to the exception so that it is printed in the logs, for the user to
see. For other components, feel free to raise whatever other exception feels
right to you; for example, `RuntimeError` would make sense for a Scrapy
version mismatch, while `ValueError` may be better if the issue is the
value of a setting.
If your requirement is a minimum Scrapy version, you may use
`scrapy.__version__` to enforce your requirement. For example:
```python
from packaging.version import parse as parse_version
import scrapy
class MyComponent:
def __init__(self):
if parse_version(scrapy.__version__) < parse_version("2.7"):
raise RuntimeError(
f"{MyComponent.__qualname__} requires Scrapy 2.7 or "
f"later, which allow defining the process_spider_output "
f"method of spider middlewares as an asynchronous "
f"generator."
)
```
### API reference
The following function can be used to create an instance of a component class:
#### scrapy.utils.misc.build_from_crawler(objcls: ~scrapy.utils.misc.SupportsFromCrawler[~scrapy.utils.misc._T_co, ~_P], crawler: Crawler, /, \*args: ~typing.~_P, \*\*kwargs: ~typing.~_P) → \_T_co
#### scrapy.utils.misc.build_from_crawler(objcls: Callable[_P, \_T_co], crawler: Crawler, /, \*args: ~typing.~_P, \*\*kwargs: ~typing.~_P) → \_T_co
Construct a class instance using its `from_crawler()` or `__init__()` constructor.
> [!WARNING]
> Added in version 2.12.
`*args` and `**kwargs` are forwarded to the constructor.
Raises `TypeError` if the resulting instance is `None`.
The following function can also be useful when implementing a component, to
report the import path of the component class, e.g. when reporting problems:
#### scrapy.utils.python.global_object_name(obj: Any) → str
Return the full import path of the given object.
```pycon
>>> from scrapy import Request
>>> global_object_name(Request)
'scrapy.http.request.Request'
>>> global_object_name(Request.replace)
'scrapy.http.request.Request.replace'
```
Source: /topics/api.md
## Core API
This section documents the Scrapy core API, and it’s intended for developers of
extensions and middlewares.
### Crawler API
The main entry point to the Scrapy API is the `Crawler`
object, which components can get for
initialization. It provides access to all Scrapy core
components, and it is the only way for components to access them and hook their
functionality into Scrapy.
The Extension Manager is responsible for loading and keeping track of installed
extensions and it’s configured through the `EXTENSIONS` setting which
contains a dictionary of all available extensions and their order similar to
how you configure the downloader middlewares.
#### *class* scrapy.crawler.Crawler(spidercls: type[Spider], settings: dict[str, Any] | Settings | None = None, init_reactor: bool = False)
The Crawler object must be instantiated with a
`scrapy.Spider` subclass and a
`scrapy.settings.Settings` object.
The `engine`, `extensions`, `logformatter`,
`request_fingerprinter` and `stats` attributes get their value
when the crawl starts, and raise `RuntimeError` when read before that.
> [!WARNING]
> Changed in version 2.18.0: Those attributes used to be `None` before getting their value.
##### request_fingerprinter
The request fingerprint builder of this crawler.
This is used from extensions and middlewares to build short, unique
identifiers for requests. See request-fingerprints.
##### settings
The settings manager of this crawler.
This is used by extensions & middlewares to access the Scrapy settings
of this crawler.
For an introduction on Scrapy settings see topics-settings.
For the API see `Settings` class.
##### signals
The signals manager of this crawler.
This is used by extensions & middlewares to hook themselves into Scrapy
functionality.
For an introduction on signals see topics-signals.
For the API see `SignalManager` class.
##### stats
The stats collector of this crawler.
This is used from extensions & middlewares to record stats of their
behaviour, or access stats collected by other extensions.
For an introduction on stats collection see topics-stats.
For the API see `StatsCollector` class.
##### logformatter *: \_LateAttribute[LogFormatter]*
The log formatter of this crawler.
This is used from extensions & middlewares to build the messages that they
log about crawling events.
For the API see the `LogFormatter` class.
##### extensions
The extension manager that keeps track of enabled extensions.
Most extensions won’t need to access this attribute.
For an introduction on extensions and a list of available extensions on
Scrapy see topics-extensions.
##### engine
The execution engine, which coordinates the core crawling logic
between the scheduler, downloader and spiders.
Some extension may want to access the Scrapy engine, to inspect or
modify the downloader and scheduler behaviour, although this is an
advanced use and this API is not yet stable.
##### spider
Spider currently being crawled. This is an instance of the spider class
provided while constructing the crawler, and it is created after the
arguments given in the `crawl()` method.
##### *async* crawl_async(\*args: Any, \*\*kwargs: Any) → None
Start the crawler by instantiating its spider class with the given
*args* and *kwargs* arguments, while setting the execution engine in
motion. Should be called only once.
> [!WARNING]
> Added in version 2.14.
Complete when the crawl is finished.
##### crawl(\*args: Any, \*\*kwargs: Any) → Generator[Deferred[Any], Any, None]
Start the crawler by instantiating its spider class with the given
*args* and *kwargs* arguments, while setting the execution engine in
motion. Should be called only once.
Return a deferred that is fired when the crawl is finished.
##### *async* stop_async() → None
Start a graceful stop of the crawler and complete when the crawler is stopped.
> [!WARNING]
> Added in version 2.14.
##### stop() → Deferred[None]
Start a graceful stop of the crawler and return a deferred that is
fired when the crawler is stopped.
##### get_addon(cls: type[\_T]) → \_T | None
Return the run-time instance of an add-on of
the specified class or a subclass, or `None` if none is found.
> [!WARNING]
> Added in version 2.12.
##### get_downloader_middleware(cls: type[\_T]) → \_T | None
Return the run-time instance of a downloader middleware of the specified class or a subclass,
or `None` if none is found.
> [!WARNING]
> Added in version 2.12.
This method can only be called after the crawl engine has been created,
e.g. at signals `engine_started` or `spider_opened`.
##### get_extension(cls: type[\_T]) → \_T | None
Return the run-time instance of an extension of the specified class or a subclass,
or `None` if none is found.
> [!WARNING]
> Added in version 2.12.
This method can only be called after the extension manager has been
created, e.g. at signals `engine_started` or
`spider_opened`.
##### get_item_pipeline(cls: type[\_T]) → \_T | None
Return the run-time instance of a item pipeline of the specified class or a subclass, or
`None` if none is found.
> [!WARNING]
> Added in version 2.12.
This method can only be called after the crawl engine has been created,
e.g. at signals `engine_started` or `spider_opened`.
##### get_spider_middleware(cls: type[\_T]) → \_T | None
Return the run-time instance of a spider middleware of the specified class or a subclass, or
`None` if none is found.
> [!WARNING]
> Added in version 2.12.
This method can only be called after the crawl engine has been created,
e.g. at signals `engine_started` or `spider_opened`.
#### *class* scrapy.crawler.AsyncCrawlerRunner(settings: dict[str, Any] | Settings | None = None)
This is a convenient helper class that keeps track of, manages and runs
crawlers inside an already setup `reactor` or
asyncio event loop.
The AsyncCrawlerRunner object must be instantiated with a
`Settings` object.
When the `TWISTED_REACTOR_ENABLED` setting is set to `True`,
this class requires a reactor to be installed and uses it, otherwise it
requires a reactor to not be installed but requires an asyncio event loop
to be installed and uses it.
This class shouldn’t be needed (since Scrapy is responsible of using it
accordingly) unless writing scripts that manually handle the crawling
process. See run-from-script for an example.
This class provides coroutine APIs. It requires
`AsyncioSelectorReactor` when used
with a reactor.
##### crawl(crawler_or_spidercls: type[Spider] | str | Crawler, \*args: Any, \*\*kwargs: Any) → Task[None]
Run a crawler with the provided arguments.
It will call the given Crawler’s `crawl_async()` method,
while keeping track of it so it can be stopped later.
If `crawler_or_spidercls` isn’t a `Crawler`
instance, this method will try to create one using this parameter as
the spider class given to it.
Returns a `Task` object which completes when the
crawling is finished.
* **Parameters:**
* **crawler_or_spidercls** (`Crawler` instance,
`Spider` subclass or string) – already created crawler, or a spider class
or spider’s name inside the project to create it
* **args** – arguments to initialize the spider
* **kwargs** – keyword arguments to initialize the spider
##### *async* stop() → None
Stops simultaneously all the crawling jobs taking place.
Completes when they all have ended.
##### *async* join() → None
Completes when all managed `crawlers` have completed their
executions.
#### *class* scrapy.crawler.CrawlerRunner(settings: dict[str, Any] | Settings | None = None)
This is a convenient helper class that keeps track of, manages and runs
crawlers inside an already setup `reactor`.
The CrawlerRunner object must be instantiated with a
`Settings` object.
This class shouldn’t be needed (since Scrapy is responsible of using it
accordingly) unless writing scripts that manually handle the crawling
process. See run-from-script for an example.
This class provides Deferred-based APIs. Use `AsyncCrawlerRunner`
for modern coroutine APIs.
##### crawl(crawler_or_spidercls: type[Spider] | str | Crawler, \*args: Any, \*\*kwargs: Any) → Deferred[None]
Run a crawler with the provided arguments.
It will call the given Crawler’s `crawl()` method, while
keeping track of it so it can be stopped later.
If `crawler_or_spidercls` isn’t a `Crawler`
instance, this method will try to create one using this parameter as
the spider class given to it.
Returns a deferred that is fired when the crawling is finished.
* **Parameters:**
* **crawler_or_spidercls** (`Crawler` instance,
`Spider` subclass or string) – already created crawler, or a spider class
or spider’s name inside the project to create it
* **args** – arguments to initialize the spider
* **kwargs** – keyword arguments to initialize the spider
##### stop() → Deferred[Any]
Stops simultaneously all the crawling jobs taking place.
Returns a deferred that is fired when they all have ended.
##### join()
Returns a deferred that is fired when all managed `crawlers` have
completed their executions.
#### *class* scrapy.crawler.AsyncCrawlerProcess(settings: dict[str, Any] | Settings | None = None, install_root_handler: bool | None = None)
Bases: `CrawlerProcessBase`, `AsyncCrawlerRunner`
A class to run multiple scrapy crawlers in a process simultaneously.
This class extends `AsyncCrawlerRunner` by adding support
for starting a `reactor` and handling shutdown
signals, like the keyboard interrupt command Ctrl-C. It also configures
top-level logging.
This utility should be a better fit than
`AsyncCrawlerRunner` if you aren’t running another
`reactor` within your application.
The AsyncCrawlerProcess object must be instantiated with a
`Settings` object.
When the `TWISTED_REACTOR_ENABLED` setting is set to `True`,
this class installs a reactor and uses it, otherwise it requires a reactor
to not be installed but installs an asyncio event loop and uses it.
* **Parameters:**
**install_root_handler** – whether to install root logging handler
(default: the `LOG_INSTALL_ROOT_HANDLER` setting)
This class shouldn’t be needed (since Scrapy is responsible of using it
accordingly) unless writing scripts that manually handle the crawling
process. See run-from-script for an example.
This class provides coroutine APIs. It requires
`AsyncioSelectorReactor` when used
with a reactor.
##### start(stop_after_crawl: bool = True, install_signal_handlers: bool = True) → None
This method starts a `reactor` or an asyncio
event loop, depending on the value of the
`TWISTED_REACTOR_ENABLED` setting.
When using a reactor it adjusts its pool size to
`REACTOR_THREADPOOL_MAXSIZE` and installs a DNS resolver based
on `TWISTED_DNS_RESOLVER`.
If `stop_after_crawl` is True, the reactor/event loop will be stopped
after all crawlers have finished, using `join()`.
* **Parameters:**
* **stop_after_crawl** (*bool*) – stop or not the reactor when all
crawlers have finished
* **install_signal_handlers** (*bool*) – whether to install the OS signal
handlers from Twisted and Scrapy (default: True)
##### crawl(crawler_or_spidercls: type[Spider] | str | Crawler, \*args: Any, \*\*kwargs: Any) → Task[None]
Run a crawler with the provided arguments.
It will call the given Crawler’s `crawl_async()` method,
while keeping track of it so it can be stopped later.
If `crawler_or_spidercls` isn’t a `Crawler`
instance, this method will try to create one using this parameter as
the spider class given to it.
Returns a `Task` object which completes when the
crawling is finished.
* **Parameters:**
* **crawler_or_spidercls** (`Crawler` instance,
`Spider` subclass or string) – already created crawler, or a spider class
or spider’s name inside the project to create it
* **args** – arguments to initialize the spider
* **kwargs** – keyword arguments to initialize the spider
##### *property* crawlers *: set[Crawler]*
Set of `crawlers` started by
`crawl()` and managed by this class.
##### create_crawler(crawler_or_spidercls: type[Spider] | str | Crawler) → Crawler
Return a `Crawler` object.
* If `crawler_or_spidercls` is a Crawler, the runner’s settings are
merged into it as defaults: for each setting, the runner’s value
is applied only if the Crawler does not already have that setting at
an equal or higher priority. The Crawler is then returned.
* If `crawler_or_spidercls` is a Spider subclass, a new Crawler
is constructed for it using this runner’s settings.
* If `crawler_or_spidercls` is a string, this function finds
a spider with this name in a Scrapy project (using spider loader),
then creates a Crawler instance for it.
##### *async* join() → None
Completes when all managed `crawlers` have completed their
executions.
##### *async* stop() → None
Stops simultaneously all the crawling jobs taking place.
Completes when they all have ended.
#### *class* scrapy.crawler.CrawlerProcess(settings: dict[str, Any] | Settings | None = None, install_root_handler: bool | None = None)
Bases: `CrawlerProcessBase`, `CrawlerRunner`
A class to run multiple scrapy crawlers in a process simultaneously.
This class extends `CrawlerRunner` by adding support
for starting a `reactor` and handling shutdown
signals, like the keyboard interrupt command Ctrl-C. It also configures
top-level logging.
This utility should be a better fit than
`CrawlerRunner` if you aren’t running another
`reactor` within your application.
The CrawlerProcess object must be instantiated with a
`Settings` object.
* **Parameters:**
**install_root_handler** – whether to install root logging handler
(default: the `LOG_INSTALL_ROOT_HANDLER` setting)
This class shouldn’t be needed (since Scrapy is responsible of using it
accordingly) unless writing scripts that manually handle the crawling
process. See run-from-script for an example.
This class provides Deferred-based APIs. Use `AsyncCrawlerProcess`
for modern coroutine APIs.
##### start(stop_after_crawl: bool = True, install_signal_handlers: bool = True) → None
This method starts a `reactor`, adjusts its pool
size to `REACTOR_THREADPOOL_MAXSIZE`, and installs a DNS
resolver based on `TWISTED_DNS_RESOLVER`.
If `stop_after_crawl` is True, the reactor will be stopped after all
crawlers have finished, using `join()`.
* **Parameters:**
* **stop_after_crawl** (*bool*) – stop or not the reactor when all
crawlers have finished
* **install_signal_handlers** (*bool*) – whether to install the OS signal
handlers from Twisted and Scrapy (default: True)
##### crawl(crawler_or_spidercls: type[Spider] | str | Crawler, \*args: Any, \*\*kwargs: Any) → Deferred[None]
Run a crawler with the provided arguments.
It will call the given Crawler’s `crawl()` method, while
keeping track of it so it can be stopped later.
If `crawler_or_spidercls` isn’t a `Crawler`
instance, this method will try to create one using this parameter as
the spider class given to it.
Returns a deferred that is fired when the crawling is finished.
* **Parameters:**
* **crawler_or_spidercls** (`Crawler` instance,
`Spider` subclass or string) – already created crawler, or a spider class
or spider’s name inside the project to create it
* **args** – arguments to initialize the spider
* **kwargs** – keyword arguments to initialize the spider
##### *property* crawlers *: set[Crawler]*
Set of `crawlers` started by
`crawl()` and managed by this class.
##### create_crawler(crawler_or_spidercls: type[Spider] | str | Crawler) → Crawler
Return a `Crawler` object.
* If `crawler_or_spidercls` is a Crawler, the runner’s settings are
merged into it as defaults: for each setting, the runner’s value
is applied only if the Crawler does not already have that setting at
an equal or higher priority. The Crawler is then returned.
* If `crawler_or_spidercls` is a Spider subclass, a new Crawler
is constructed for it using this runner’s settings.
* If `crawler_or_spidercls` is a string, this function finds
a spider with this name in a Scrapy project (using spider loader),
then creates a Crawler instance for it.
##### join()
Returns a deferred that is fired when all managed `crawlers` have
completed their executions.
##### stop() → Deferred[Any]
Stops simultaneously all the crawling jobs taking place.
Returns a deferred that is fired when they all have ended.
### Settings API
#### scrapy.settings.SETTINGS_PRIORITIES
Dictionary that sets the key name and priority level of the default
settings priorities used in Scrapy.
Each item defines a settings entry point, giving it a code name for
identification and an integer priority. Greater priorities take more
precedence over lesser ones when setting and retrieving values in the
`Settings` class.
```python
SETTINGS_PRIORITIES = {
"default": 0,
"command": 10,
"addon": 15,
"project": 20,
"spider": 30,
"cmdline": 40,
}
```
For a detailed explanation on each settings sources, see:
topics-settings.
#### scrapy.settings.get_settings_priority(priority: int | str) → int
Small helper function that looks up a given string priority in the
`SETTINGS_PRIORITIES` dictionary and returns its
numerical value, or directly returns a given numerical priority.
#### scrapy.utils.project.get_project_settings() → Settings
Return a `Settings` object with the
project settings of the current project.
Settings from sources with a higher precedence, such as spider
settings, are applied when a crawl starts.
#### *class* scrapy.settings.Settings(values: \_SettingsInput = None, priority: int | str = 'project')
Bases: `BaseSettings`
This object stores Scrapy settings for the configuration of internal
components, and can be used for any further customization.
It is a direct subclass and supports all methods of
`BaseSettings`. Additionally, after instantiation
of this class, the new object will have the global default settings
described on topics-settings-ref already populated.
#### *class* scrapy.settings.BaseSettings(values: \_SettingsInput = None, priority: int | str = 'project')
Instances of this class behave like dictionaries, but store priorities
along with their `(key, value)` pairs, and can be frozen (i.e. marked
immutable).
Key-value entries can be passed on initialization with the `values`
argument, and they would take the `priority` level (unless `values` is
already an instance of `BaseSettings`, in which
case the existing priority levels will be kept). If the `priority`
argument is a string, the priority name will be looked up in
`SETTINGS_PRIORITIES`. Otherwise, a specific integer
should be provided.
Once the object is created, new settings can be loaded or updated with the
`set()` method, and can be accessed with
the square bracket notation of dictionaries, or with the
`get()` method of the instance and its
value conversion variants. When requesting a stored key, the value with the
highest priority will be retrieved.
##### add_to_list(name: str, item: Any) → None
Append *item* to the `list` setting with the specified *name*
if *item* is not already in that list.
This change is applied regardless of the priority of the *name*
setting. The setting priority is not affected by this change either.
##### remove_from_list(name: str, item: Any) → None
Remove *item* from the `list` setting with the specified
*name*.
If *item* is missing, raise `ValueError`.
This change is applied regardless of the priority of the *name*
setting. The setting priority is not affected by this change either.
##### get(name: str, default: Any = None) → Any
Get a setting value without affecting its original type.
* **Parameters:**
* **name** (*str*) – the setting name
* **default** (*object*) – the value to return if no setting is found
##### getbool(name: str, default: bool = False) → bool
Get a setting value as a boolean.
`1`, `'1'`, True\` and `'True'` return `True`,
while `0`, `'0'`, `False`, `'False'` and `None` return `False`.
For example, settings populated through environment variables set to
`'0'` will return `False` when using this method.
* **Parameters:**
* **name** (*str*) – the setting name
* **default** (*object*) – the value to return if no setting is found
##### getint(name: str, default: int = 0) → int
Get a setting value as an int.
* **Parameters:**
* **name** (*str*) – the setting name
* **default** (*object*) – the value to return if no setting is found
##### getfloat(name: str, default: float = 0.0) → float
Get a setting value as a float.
* **Parameters:**
* **name** (*str*) – the setting name
* **default** (*object*) – the value to return if no setting is found
##### getlist(name: str, default: list[Any] | None = None) → list[Any]
Get a setting value as a list. If the setting original type is a list,
a copy of it will be returned. If it’s a string it will be split by
“,”. If it is an empty string, an empty list will be returned.
For example, settings populated through environment variables set to
`'one,two'` will return a list [‘one’, ‘two’] when using this method.
* **Parameters:**
* **name** (*str*) – the setting name
* **default** (*object*) – the value to return if no setting is found
##### getdict(name: str, default: dict[Any, Any] | None = None) → dict[Any, Any]
Get a setting value as a dictionary. If the setting original type is a
dictionary, a copy of it will be returned. If it is a string it will be
evaluated as a JSON dictionary. In the case that it is a
`BaseSettings` instance itself, it will be
converted to a dictionary, containing all its current settings values
as they would be returned by `get()`,
and losing all information about priority and mutability.
* **Parameters:**
* **name** (*str*) – the setting name
* **default** (*object*) – the value to return if no setting is found
##### getdictorlist(name: str, default: dict[Any, Any] | list[Any] | tuple[Any] | None = None) → dict[Any, Any] | list[Any]
Get a setting value as either a `dict` or a `list`.
If the setting is already a dict or a list, a copy of it will be
returned.
If it is a string it will be evaluated as JSON, or as a comma-separated
list of strings as a fallback.
For example, settings populated from the command line will return:
- `{'key1': 'value1', 'key2': 'value2'}` if set to
`'{"key1": "value1", "key2": "value2"}'`
- `['one', 'two']` if set to `'["one", "two"]'` or `'one,two'`
* **Parameters:**
* **name** (*str*) – the setting name
* **default** (*object*) – the value to return if no setting is found
##### getwithbase(name: str) → BaseSettings
Get a composition of a dictionary-like setting and its `_BASE`
counterpart.
Use
`get_component_priority_dict_with_base()`
instead if the setting is a component priority dictionary.
* **Parameters:**
**name** (*str*) – name of the dictionary-like setting
##### get_component_priority_dict_with_base(name: str) → BaseSettings
Get a composition of a component priority dictionary setting and
its `_BASE` counterpart.
Keys are resolved to their import path for deduplication and then
restored to their latest input representation.
* **Parameters:**
**name** (*str*) – name of the component priority dictionary setting
##### getpriority(name: str) → int | None
Return the current numerical priority value of a setting, or `None` if
the given `name` does not exist.
* **Parameters:**
**name** (*str*) – the setting name
##### maxpriority() → int
Return the numerical value of the highest priority present throughout
all settings, or the numerical value for `default` from
`SETTINGS_PRIORITIES` if there are no settings
stored.
##### replace_in_component_priority_dict(name: str, old_cls: type, new_cls: type, priority: int | None = None) → None
Replace *old_cls* with *new_cls* in the *name* component
priority dictionary.
If *old_cls* is missing, or has `None` as value, `KeyError`
is raised.
If *old_cls* was present as an import string, even more than once,
those keys are dropped and replaced by *new_cls*.
If *priority* is specified, that is the value assigned to *new_cls* in
the component priority dictionary. Otherwise, the value of *old_cls* is
used. If *old_cls* was present multiple times (possible with import
strings) with different values, the value assigned to *new_cls* is one
of them, with no guarantee about which one it is.
This change is applied regardless of the priority of the *name*
setting. The setting priority is not affected by this change either.
##### set(name: str, value: Any, priority: int | str = 'project') → None
Store a key/value attribute with a given priority.
Settings should be populated *before* the Crawler object applies them
(in the `crawl_async()` or
`crawl()` method), otherwise they won’t
have any effect.
* **Parameters:**
* **name** (*str*) – the setting name
* **value** (*object*) – the value to associate with the setting
* **priority** (*str* *or* *int*) – the priority of the setting. Should be a key of
`SETTINGS_PRIORITIES` or an integer
##### set_in_component_priority_dict(name: str, cls: type, priority: int | None) → None
Set the *cls* component in the *name* component priority
dictionary setting with *priority*.
If *cls* already exists, its value is updated.
If *cls* was present as an import string, even more than once, those
keys are dropped and replaced by *cls*.
This change is applied regardless of the priority of the *name*
setting. The setting priority is not affected by this change either.
##### setdefault([d]k,) → D.get(k,d), also set D[k]=d if k not in D
##### setdefault_in_component_priority_dict(name: str, cls: type, priority: int | None) → None
Set the *cls* component in the *name* component priority
dictionary setting with *priority*
if not already defined (even as an import string).
If *cls* is not already defined, it is set regardless of the priority
of the *name* setting. The setting priority is not affected by this
change either.
##### setmodule(module: ModuleType | str, priority: int | str = 'project') → None
Store settings from a module with a given priority.
This is a helper function that calls
`set()` for every globally declared
uppercase variable of `module` with the provided `priority`.
* **Parameters:**
* **module** (*types.ModuleType* *or* *str*) – the module or the path of the module
* **priority** (*str* *or* *int*) – the priority of the settings. Should be a key of
`SETTINGS_PRIORITIES` or an integer
##### update(values: \_SettingsInput, priority: int | str = 'project') → None
Store key/value pairs with a given priority.
This is a helper function that calls
`set()` for every item of `values`
with the provided `priority`.
If `values` is a string, it is assumed to be JSON-encoded and parsed
into a dict with `json.loads()` first. If it is a
`BaseSettings` instance, the per-key priorities
will be used and the `priority` parameter ignored. This allows
inserting/updating settings with different priorities with a single
command.
* **Parameters:**
* **values** (dict, iterable, string or `BaseSettings`) – the settings names and values
* **priority** (*str* *or* *int*) – the priority of the settings. Should be a key of
`SETTINGS_PRIORITIES` or an integer
##### copy() → Self
Make a deep copy of current settings.
This method returns a new instance of this class,
populated with the same values and their priorities.
Modifications to the new object won’t be reflected on the original
settings.
##### freeze() → None
Disable further changes to the current settings.
After calling this method, the present state of the settings will become
immutable. Trying to change values through the `set()` method and
its variants won’t be possible and will be alerted.
##### frozencopy() → Self
Return an immutable copy of the current settings.
Alias for a `freeze()` call in the object returned by `copy()`.
##### copy_to_dict() → dict[str, Any]
Make a copy of current settings and convert to a dict.
This method returns a new dict populated with the same values
as the current settings.
Modifications to the returned dict won’t be reflected on the original
settings.
This method can be useful for example for printing settings
in Scrapy shell.
##### pop([d]k,) → v, remove specified key and return the corresponding
value. If key is not found, d is returned if given, otherwise
KeyError is raised.
### SpiderLoader API
Custom spider loaders can be employed by specifying their path in the
`SPIDER_LOADER_CLASS` project setting. They must implement
`SpiderLoaderProtocol`.
#### *class* scrapy.spiderloader.SpiderLoaderProtocol(\*args, \*\*kwargs)
Protocol for spider loader implementations.
See `SPIDER_LOADER_CLASS`.
##### *classmethod* from_settings(settings: BaseSettings) → Self
Return an instance of the class for the given settings.
##### load(spider_name: str) → type[Spider]
Return the spider class for the given spider name. If the spider
name is not found, it must raise a `KeyError`.
##### list() → list[str]
Return a list with the names of all spiders available in the
project.
##### find_by_request(request: Request) → \_\_builtins_\_.list[str]
Return the list of spiders names that can handle the given request.
#### *class* scrapy.spiderloader.SpiderLoader(settings: BaseSettings)
SpiderLoader is a class which locates and loads spiders
in a Scrapy project.
##### *classmethod* from_settings(settings: BaseSettings) → Self
Create an instance of the class.
It’s called with the current project settings, and it loads the spiders
found recursively in the modules of the `SPIDER_MODULES`
setting.
##### load(spider_name: str) → type[Spider]
Return the spider class for the given spider name.
If the spider name is not found, raise a `KeyError`.
##### find_by_request(request: Request) → list[str]
Return the list of spider names that can handle the given request.
It will try to match the request’s url against the domains of
the spiders.
##### list() → list[str]
Return a list with the names of all spiders available in the project.
#### *class* scrapy.spiderloader.DummySpiderLoader
A dummy spider loader that does not load any spiders.
### Signals API
#### *class* scrapy.signalmanager.SignalManager(sender: Any = \_Anonymous)
##### connect(receiver: Any, signal: Any, \*\*kwargs: Any) → None
Connect a receiver function to a signal.
The signal can be any object, although Scrapy comes with some
predefined signals that are documented in the topics-signals
section.
* **Parameters:**
* **receiver** (*collections.abc.Callable*) – the function to be connected
* **signal** (*object*) – the signal to connect to
##### disconnect(receiver: Any, signal: Any, \*\*kwargs: Any) → None
Disconnect a receiver function from a signal. This has the
opposite effect of the `connect()` method, and the arguments
are the same.
##### send_catch_log(signal: Any, \*\*kwargs: Any) → list[tuple[Any, Any]]
Send a signal, catch exceptions and log them.
The keyword arguments are passed to the signal handlers (connected
through the `connect()` method).
##### send_catch_log_deferred(signal: Any, \*\*kwargs: Any) → Deferred[list[tuple[Any, Any]]]
Like `send_catch_log()` but supports asynchronous signal
handlers.
Returns a Deferred that gets fired once all signal handlers
have finished. Send a signal, catch exceptions and log them.
The keyword arguments are passed to the signal handlers (connected
through the `connect()` method).
##### *async* send_catch_log_async(signal: Any, \*\*kwargs: Any) → list[tuple[Any, Any]]
Like `send_catch_log()` but supports asynchronous signal
handlers.
Returns a coroutine that completes once all signal handlers
have finished. Send a signal, catch exceptions and log them.
The keyword arguments are passed to the signal handlers (connected
through the `connect()` method).
> [!WARNING]
> Added in version 2.14.
##### disconnect_all(signal: Any, \*\*kwargs: Any) → None
Disconnect all receivers from the given signal.
* **Parameters:**
**signal** (*object*) – the signal to disconnect from
##### *async* wait_for(signal: Any) → None
Await the next *signal*.
See start-requests-lazy for an example.
### Stats Collector API
There are several Stats Collectors available under the
`scrapy.statscollectors` module and they all implement the Stats
Collector API defined by the `StatsCollector`
class (which they all inherit from).
#### *class* scrapy.statscollectors.StatsCollector
##### get_value(key, default=None)
Return the value for the given stats key or default if it doesn’t exist.
##### get_stats()
Get all stats from the currently running spider as a dict.
##### set_value(key, value)
Set the given value for the given stats key.
##### set_stats(stats)
Override the current stats with the dict passed in `stats` argument.
##### inc_value(key, count=1, start=0)
Increment the value of the given stats key, by the given count,
assuming the start value given (when it’s not set).
##### max_value(key, value)
Set the given value for the given key only if current value for the
same key is lower than value. If there is no current value for the
given key, the value is always set.
##### min_value(key, value)
Set the given value for the given key only if current value for the
same key is greater than value. If there is no current value for the
given key, the value is always set.
##### clear_stats()
Clear all stats.
The following methods are not part of the stats collection api but instead
used when implementing custom stats collectors:
##### open_spider()
Open the spider for stats collection.
##### close_spider()
Close the spider. After this is called, no more specific stats
can be accessed or collected.
### Engine API
#### *class* scrapy.core.engine.ExecutionEngine
##### *property* scheduler *: BaseScheduler | None*
The scheduler in use, or `None` before the spider has started.
> [!WARNING]
> Added in version 2.19.0.
##### needs_backout() → bool
Returns `True` if no more requests can be sent at the moment, or
`False` otherwise.
See start-requests-lazy for an example.
Source: /versioning.md
## Versioning and API stability
### Versioning
There are 3 numbers in a Scrapy version: *A.B.C*
* *A* is the major version. This will rarely change and will signify very
large changes.
* *B* is the release number. This will include many changes including features
and things that possibly break backward compatibility, although we strive to
keep these cases at a minimum.
* *C* is the bugfix release number.
Backward-incompatibilities are explicitly mentioned in the release notes,
and may require special attention before upgrading.
Development releases do not follow 3-numbers version and are generally
released as `dev` suffixed versions, e.g. `1.3dev`.
> [!NOTE]
> With Scrapy 0.\* series, Scrapy used odd-numbered versions for development releases.
> This is not the case anymore from Scrapy 1.0 onwards.
>
> Starting with Scrapy 1.0, all releases should be considered production-ready.
For example:
* *1.1.1* is the first bugfix release of the *1.1* series (safe to use in
production)
### API stability
API stability was one of the major goals for the *1.0* release.
Methods or functions that start with a single underscore (`_`) are private
and should never be relied upon as stable.
Also, keep in mind that stable doesn’t mean complete: stable APIs could grow
new methods or functionality but the existing methods should keep working the
same way.
### Deprecation policy
We aim to maintain support for deprecated Scrapy features for at least 1 year.
For example, if a feature is deprecated in a Scrapy version released on
June 15th 2020, that feature should continue to work in versions released on
June 14th 2021 or before that.
Any new Scrapy release after a year *may* remove support for that deprecated
feature.
All deprecated features removed in a Scrapy release are explicitly mentioned in
the release notes.
