<a id="topics-downloader-middleware"></a>

# Downloader Middleware

The downloader middleware is a framework of hooks into Scrapy’s
request/response processing.  It’s a light, low-level system for globally
altering Scrapy’s requests and responses at the HTTP level. See
[Scrapy’s building blocks](concepts.md#concepts) for a rundown of other alternatives.

<a id="topics-downloader-middleware-setting"></a>

## Activating a downloader middleware

To activate a downloader middleware component, add it to the
[`DOWNLOADER_MIDDLEWARES`](settings.md#std-setting-DOWNLOADER_MIDDLEWARES) setting, which is a dict whose keys are the
middleware class paths and their values are the middleware orders.

Here’s an example:

```python
DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.CustomDownloaderMiddleware": 543,
}
```

The [`DOWNLOADER_MIDDLEWARES`](settings.md#std-setting-DOWNLOADER_MIDDLEWARES) setting is merged with the
[`DOWNLOADER_MIDDLEWARES_BASE`](settings.md#std-setting-DOWNLOADER_MIDDLEWARES_BASE) setting defined in Scrapy (and not meant
to be overridden) and then sorted by order to get the final sorted list of
enabled middlewares: the first middleware is the one closer to the engine and
the last is the one closer to the downloader. In other words,
the [`process_request()`](#scrapy.downloadermiddlewares.DownloaderMiddleware.process_request)
method of each middleware will be invoked in increasing
middleware order (100, 200, 300, …) and the [`process_response()`](#scrapy.downloadermiddlewares.DownloaderMiddleware.process_response) method
of each middleware will be invoked in decreasing order.

To decide which order to assign to your middleware see the
[`DOWNLOADER_MIDDLEWARES_BASE`](settings.md#std-setting-DOWNLOADER_MIDDLEWARES_BASE) setting and pick a value according to
where you want to insert the middleware. The order does matter because each
middleware performs a different action and your middleware could depend on some
previous (or subsequent) middleware being applied.

If you want to disable a built-in middleware (the ones defined in
[`DOWNLOADER_MIDDLEWARES_BASE`](settings.md#std-setting-DOWNLOADER_MIDDLEWARES_BASE) and enabled by default) you must define it
in your project’s [`DOWNLOADER_MIDDLEWARES`](settings.md#std-setting-DOWNLOADER_MIDDLEWARES) setting and assign `None`
as its value.  For example, if you want to disable the user-agent middleware:

```python
DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.CustomDownloaderMiddleware": 543,
    "scrapy.downloadermiddlewares.useragent.UserAgentMiddleware": None,
}
```

Finally, keep in mind that some middlewares may need to be enabled through a
particular setting. See each middleware documentation for more info.

<a id="topics-downloader-middleware-custom"></a>

## Writing your own downloader middleware

Each downloader middleware is a [component](components.md#topics-components) that
defines one or more of these methods:

<a id="module-scrapy.downloadermiddlewares"></a>

### *class* scrapy.downloadermiddlewares.DownloaderMiddleware

> [!NOTE]
> Any of the downloader middleware methods may be defined as a
> coroutine function (`async def`).

#### process_request(request)

This method is called for each request that goes through the download
middleware.

[`process_request()`](#scrapy.downloadermiddlewares.DownloaderMiddleware.process_request) should either: return `None`, return a
[`Response`](request-response.md#scrapy.http.Response) object, return a [`Request`](request-response.md#scrapy.Request)
object, or raise [`IgnoreRequest`](exceptions.md#scrapy.exceptions.IgnoreRequest).

If it returns `None`, Scrapy will continue processing this request, executing all
other middlewares until, finally, the appropriate downloader handler is called
the request performed (and its response downloaded).

If it returns a [`Response`](request-response.md#scrapy.http.Response) object, Scrapy won’t bother
calling *any* other [`process_request()`](#scrapy.downloadermiddlewares.DownloaderMiddleware.process_request) or [`process_exception()`](#scrapy.downloadermiddlewares.DownloaderMiddleware.process_exception) methods,
or the appropriate download function; it’ll return that response. The [`process_response()`](#scrapy.downloadermiddlewares.DownloaderMiddleware.process_response)
methods of installed middleware is always called on every response.

If it returns a [`Request`](request-response.md#scrapy.Request) object, Scrapy will stop calling
[`process_request()`](#scrapy.downloadermiddlewares.DownloaderMiddleware.process_request) methods and reschedule the returned request. Once the newly returned
request is performed, the appropriate middleware chain will be called on
the downloaded response.

If it raises an [`IgnoreRequest`](exceptions.md#scrapy.exceptions.IgnoreRequest) exception, the
[`process_exception()`](#scrapy.downloadermiddlewares.DownloaderMiddleware.process_exception) methods of installed downloader middleware will be called.
If none of them handle the exception, the errback function of the request
(`Request.errback`) is called. If no code handles the raised exception, it is
ignored and not logged (unlike other exceptions).

* **Parameters:**
  **request** ([`Request`](request-response.md#scrapy.Request) object) – the request being processed

#### process_response(request, response)

[`process_response()`](#scrapy.downloadermiddlewares.DownloaderMiddleware.process_response) should either: return a [`Response`](request-response.md#scrapy.http.Response)
object, return a [`Request`](request-response.md#scrapy.Request) object or
raise a [`IgnoreRequest`](exceptions.md#scrapy.exceptions.IgnoreRequest) exception.

If it returns a [`Response`](request-response.md#scrapy.http.Response) (it could be the same given
response, or a brand-new one), that response will continue to be processed
with the [`process_response()`](#scrapy.downloadermiddlewares.DownloaderMiddleware.process_response) of the next middleware in the chain.

If it returns a [`Request`](request-response.md#scrapy.Request) object, the middleware chain is
halted and the returned request is rescheduled to be downloaded in the future.
This is the same behavior as if a request is returned from [`process_request()`](#scrapy.downloadermiddlewares.DownloaderMiddleware.process_request).

If it raises an [`IgnoreRequest`](exceptions.md#scrapy.exceptions.IgnoreRequest) exception, the errback
function of the request (`Request.errback`) is called. If no code handles the raised
exception, it is ignored and not logged (unlike other exceptions).

* **Parameters:**
  * **request** (is a [`Request`](request-response.md#scrapy.Request) object) – the request that originated the response
  * **response** ([`Response`](request-response.md#scrapy.http.Response) object) – the response being processed

#### process_exception(request, exception)

Scrapy calls [`process_exception()`](#scrapy.downloadermiddlewares.DownloaderMiddleware.process_exception) when a [download handler](download-handlers.md#topics-download-handlers) or a [`process_request()`](#scrapy.downloadermiddlewares.DownloaderMiddleware.process_request) (from a
downloader middleware) raises an exception (including an
[`IgnoreRequest`](exceptions.md#scrapy.exceptions.IgnoreRequest) exception).

[`process_exception()`](#scrapy.downloadermiddlewares.DownloaderMiddleware.process_exception) should return: either `None`,
a [`Response`](request-response.md#scrapy.http.Response) object, or a [`Request`](request-response.md#scrapy.Request) object.

If it returns `None`, Scrapy will continue processing this exception,
executing any other [`process_exception()`](#scrapy.downloadermiddlewares.DownloaderMiddleware.process_exception) methods of installed middleware,
until no middleware is left and the default exception handling kicks in.

If it returns a [`Response`](request-response.md#scrapy.http.Response) object, the [`process_response()`](#scrapy.downloadermiddlewares.DownloaderMiddleware.process_response)
method chain of installed middleware is started, and Scrapy won’t bother calling
any other [`process_exception()`](#scrapy.downloadermiddlewares.DownloaderMiddleware.process_exception) methods of middleware.

If it returns a [`Request`](request-response.md#scrapy.Request) object, the returned request is
rescheduled to be downloaded in the future. This stops the execution of
[`process_exception()`](#scrapy.downloadermiddlewares.DownloaderMiddleware.process_exception) methods of the middleware the same as returning a
response would.

* **Parameters:**
  * **request** (is a [`Request`](request-response.md#scrapy.Request) object) – the request that generated the exception
  * **exception** (an `Exception` object) – the raised exception

<a id="mw-download"></a>

## Downloading a request from a downloader middleware

A downloader middleware can download a request of its own while it processes
another one, e.g. to fetch something that the request it is processing needs.
The built-in [robots.txt middleware](#topics-dlmw-robots) does that: it
holds each request while it downloads the `robots.txt` file of its website.

Use `crawler.engine.download_async()` for that:

```python
from scrapy import Request
from scrapy.http.request import NO_CALLBACK


class TokenMiddleware:
    def __init__(self, crawler):
        self.crawler = crawler
        self.token = None

    @classmethod
    def from_crawler(cls, crawler):
        return cls(crawler)

    async def process_request(self, request):
        if request.meta.get("dont_obey_robotstxt"):
            return
        if self.token is None:
            response = await self.crawler.engine.download_async(
                Request(
                    "https://example.com/token",
                    callback=NO_CALLBACK,
                    meta={"dont_obey_robotstxt": True},
                )
            )
            self.token = response.text
        request.headers["Authorization"] = self.token
```

Requests that you download this way go through the downloader middleware chain
as well, including your own middleware and the [robots.txt middleware](#topics-dlmw-robots), which holds a request until the `robots.txt` file of
its website arrives. Be careful not to introduce deadlocks: a request that you
download must not end up waiting for the request that is waiting for it. Hence
[`dont_obey_robotstxt`](#std-reqmeta-dont_obey_robotstxt) above, which makes both middlewares let the token
request through.

While the first token response is in transit, `process_request` runs for other
requests as well, and the middleware above downloads a token for each of them.
Cache the task that downloads the token, and not only its result, to download
the token only once.

<a id="topics-downloader-middleware-ref"></a>

## Built-in downloader middleware reference

This page describes all downloader middleware components that come with
Scrapy. For information on how to use them and how to write your own downloader
middleware, see the [downloader middleware usage guide](#topics-downloader-middleware).

For a list of the components enabled by default (and their orders) see the
[`DOWNLOADER_MIDDLEWARES_BASE`](settings.md#std-setting-DOWNLOADER_MIDDLEWARES_BASE) setting.

### CookiesMiddleware

See [Cookies](cookies.md#cookies).

### DefaultHeadersMiddleware

### *class* scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware

This middleware sets all default requests headers specified in the
[`DEFAULT_REQUEST_HEADERS`](settings.md#std-setting-DEFAULT_REQUEST_HEADERS) setting.

### DownloadTimeoutMiddleware

### *class* scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware

This middleware sets the download timeout for requests specified in the
[`DOWNLOAD_TIMEOUT`](settings.md#std-setting-DOWNLOAD_TIMEOUT) setting.

> [!NOTE]
> You can also set download timeout per-request using the
> [`download_timeout`](request-response.md#std-reqmeta-download_timeout) [`Request.meta`](request-response.md#scrapy.Request.meta) key; this is supported
> even when DownloadTimeoutMiddleware is disabled.

### HttpAuthMiddleware

### *class* scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware

This middleware authenticates requests using [Basic access authentication](https://en.wikipedia.org/wiki/Basic_access_authentication)
(aka. HTTP auth).

Use the [`HTTPAUTH_USER`](#std-setting-HTTPAUTH_USER), [`HTTPAUTH_PASS`](#std-setting-HTTPAUTH_PASS), and
[`HTTPAUTH_DOMAIN`](#std-setting-HTTPAUTH_DOMAIN) settings to configure it. You can also override
the credentials per request via [`meta`](request-response.md#scrapy.Request.meta) keys
[`http_user`](request-response.md#std-reqmeta-http_user), [`http_pass`](request-response.md#std-reqmeta-http_pass), and [`http_auth_domain`](request-response.md#std-reqmeta-http_auth_domain).

Example using settings (e.g. in [`custom_settings`](spiders.md#scrapy.Spider.custom_settings)):

```python
from scrapy.spiders import CrawlSpider


class SomeIntranetSiteSpider(CrawlSpider):
    name = "intranet.example.com"
    custom_settings = {
        "HTTPAUTH_USER": "someuser",
        "HTTPAUTH_PASS": "somepass",
        "HTTPAUTH_DOMAIN": "intranet.example.com",
    }

    # .. rest of the spider code omitted ...
```

Example using per-request meta:

```python
async def start(self):
    yield Request(
        "https://intranet.example.com/protected/",
        meta={
            "http_user": "someuser",
            "http_pass": "somepass",
            "http_auth_domain": "intranet.example.com",
        },
    )
```

<a id="std-setting-HTTPAUTH_USER"></a>

#### HTTPAUTH_USER

> [!WARNING]
> Added in version 2.17.0.

Default: `""`

The username to use for HTTP basic authentication, applied to all requests
whose URL matches [`HTTPAUTH_DOMAIN`](#std-setting-HTTPAUTH_DOMAIN).

<a id="std-setting-HTTPAUTH_PASS"></a>

#### HTTPAUTH_PASS

> [!WARNING]
> Added in version 2.17.0.

Default: `""`

The password to use for HTTP basic authentication.

<a id="std-setting-HTTPAUTH_DOMAIN"></a>

#### HTTPAUTH_DOMAIN

> [!WARNING]
> Added in version 2.17.0.

Default: `None`

The domain (and its subdomains) to which HTTP basic authentication credentials
are sent. Set to `None` to send credentials with all requests, but be aware
that this risks leaking credentials to unrelated domains.

This setting must be explicitly configured whenever [`HTTPAUTH_USER`](#std-setting-HTTPAUTH_USER)
or [`HTTPAUTH_PASS`](#std-setting-HTTPAUTH_PASS) is set.

> [!NOTE]
> ### See also
> 
> [Credential leakage across domains](security.md#security-credential-leakage)

### HttpCacheMiddleware

### *class* scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware

This middleware provides low-level cache to all HTTP requests and responses.
It has to be combined with a cache storage backend as well as a cache policy.

Scrapy ships with the following HTTP cache storage backends:

> * [Filesystem storage backend (default)](#httpcache-storage-fs)
> * [DBM storage backend](#httpcache-storage-dbm)

You can change the HTTP cache storage backend with the [`HTTPCACHE_STORAGE`](#std-setting-HTTPCACHE_STORAGE)
setting. Or you can also [implement your own storage backend.](#httpcache-storage-custom)

Scrapy ships with two HTTP cache policies:

> * [RFC2616 policy](#httpcache-policy-rfc2616)
> * [Dummy policy (default)](#httpcache-policy-dummy)

You can change the HTTP cache policy with the [`HTTPCACHE_POLICY`](#std-setting-HTTPCACHE_POLICY)
setting. Or you can also implement your own policy.

<a id="std-reqmeta-dont_cache"></a>

You can also avoid caching a response on every policy using [`dont_cache`](#std-reqmeta-dont_cache) meta key equals `True`.

<a id="std-reqmeta-cache_timestamp"></a>
> [!WARNING]
> Added in version 2.19.0.

When a response comes from the cache, its [`meta`](request-response.md#scrapy.http.Response.meta)
exposes the [`cache_timestamp`](#std-reqmeta-cache_timestamp) key: the Unix timestamp of when the
response was stored, useful e.g. to interpret dates relative to the time
the page was downloaded rather than to the current time.

<a id="httpcache-policy-dummy"></a>

#### Dummy policy (default)

### *class* scrapy.extensions.httpcache.DummyPolicy

This policy has no awareness of any HTTP Cache-Control directives.
Every request and its corresponding response are cached.  When the same
request is seen again, the response is returned without transferring
anything from the Internet.

The Dummy policy is useful for testing spiders faster (without having
to wait for downloads every time) and for trying your spider offline,
when an Internet connection is not available. The goal is to be able to
“replay” a spider run *exactly as it ran before*.

<a id="httpcache-policy-rfc2616"></a>

#### RFC2616 policy

### *class* scrapy.extensions.httpcache.RFC2616Policy

This policy provides a RFC2616 compliant HTTP cache, i.e. with HTTP
Cache-Control awareness, aimed at production and used in continuous
runs to avoid downloading unmodified data (to save bandwidth and speed up
crawls).

What is implemented:

* Do not attempt to store responses/requests with `no-store` cache-control directive set
* Do not serve responses from cache if `no-cache` cache-control directive is set even for fresh responses
* Compute freshness lifetime from `max-age` cache-control directive
* Compute freshness lifetime from `Expires` response header
* Compute freshness lifetime from `Last-Modified` response header (heuristic used by Firefox)
* Compute current age from `Age` response header
* Compute current age from `Date` header
* Revalidate stale responses based on `Last-Modified` response header
* Revalidate stale responses based on `ETag` response header
* Set `Date` header for any received response missing it
* Support `max-stale` cache-control directive in requests

This allows spiders to be configured with the full RFC2616 cache policy,
but avoid revalidation on a request-by-request basis, while remaining
conformant with the HTTP spec.

Example:

Add `Cache-Control: max-stale=600` to Request headers to accept responses that
have exceeded their expiration time by no more than 600 seconds.

See also: RFC2616, 14.9.3

What is missing:

* `Pragma: no-cache` support [https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.9.1](https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.9.1)
* `Vary` header support [https://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.6](https://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.6)
* Invalidation after updates or deletes [https://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.10](https://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.10)
* … probably others ..

<a id="httpcache-storage-fs"></a>

#### Filesystem storage backend (default)

### *class* scrapy.extensions.httpcache.FilesystemCacheStorage

File system storage backend is available for the HTTP cache middleware.

Each request/response pair is stored in a different directory containing
the following files:

* `request_body` - the plain request body
* `request_headers` - the request headers (in raw HTTP format)
* `response_body` - the plain response body
* `response_headers` - the response headers (in raw HTTP format)
* `response_data` - the remaining data of the response, as returned by
  [`Response.to_dict()`](request-response.md#scrapy.http.Response.to_dict), pickled
* `meta` - some metadata of this cache resource in Python `repr()`
  format (grep-friendly format)
* `pickled_meta` - the same metadata in `meta` but pickled for more
  efficient deserialization

The directory name is made from the request fingerprint (see
`scrapy.utils.request.fingerprint`), and one level of subdirectories is
used to avoid creating too many files into the same directory (which is
inefficient in many file systems). An example directory could be:

```
/path/to/cache/dir/example.com/72/72811f648e718090f041317756c03adb0ada46c7
```

<a id="httpcache-storage-dbm"></a>

#### DBM storage backend

### *class* scrapy.extensions.httpcache.DbmCacheStorage

A [DBM](https://en.wikipedia.org/wiki/Dbm) storage backend is also available for the HTTP cache middleware.

By default, it uses the [`dbm`](https://docs.python.org/3/library/dbm.html#module-dbm), but you can change it with the
[`HTTPCACHE_DBM_MODULE`](#std-setting-HTTPCACHE_DBM_MODULE) setting.

<a id="httpcache-storage-custom"></a>

#### Writing your own storage backend

You can implement a cache storage backend by creating a Python class that
defines the methods described below.

To store a response, use [`Response.to_dict()`](request-response.md#scrapy.http.Response.to_dict), and to read it back, use
[`response_from_dict()`](request-response.md#scrapy.utils.response.response_from_dict). That way responses of any
class, including those of third-party plugins, are cached and restored intact.

<a id="module-scrapy.extensions.httpcache"></a>

### *class* scrapy.extensions.httpcache.CacheStorage

#### open_spider(spider)

This method gets called after a spider has been opened for crawling. It handles
the [`spider_opened`](signals.md#std-signal-spider_opened) signal.

* **Parameters:**
  **spider** ([`Spider`](spiders.md#scrapy.Spider) object) – the spider which has been opened

#### close_spider(spider)

This method gets called after a spider has been closed. It handles
the [`spider_closed`](signals.md#std-signal-spider_closed) signal.

* **Parameters:**
  **spider** ([`Spider`](spiders.md#scrapy.Spider) object) – the spider which has been closed

#### retrieve_response(spider, request)

Return response if present in cache, or `None` otherwise.

If this method raises an exception, e.g. because the cache entry is
corrupted, the middleware logs a warning and handles the request as a
cache miss.

* **Parameters:**
  * **spider** ([`Spider`](spiders.md#scrapy.Spider) object) – the spider which generated the request
  * **request** ([`Request`](request-response.md#scrapy.Request) object) – the request to find cached response for

On a cache hit, set `request.meta["cache_timestamp"]` (see
[`cache_timestamp`](#std-reqmeta-cache_timestamp)) to the Unix timestamp of when the response
was stored.

#### store_response(spider, request, response)

Store the given response in the cache.

* **Parameters:**
  * **spider** ([`Spider`](spiders.md#scrapy.Spider) object) – the spider for which the response is intended
  * **request** ([`Request`](request-response.md#scrapy.Request) object) – the corresponding request the spider generated
  * **response** ([`Response`](request-response.md#scrapy.http.Response) object) – the response to store in the cache

In order to use your storage backend, set:

* [`HTTPCACHE_STORAGE`](#std-setting-HTTPCACHE_STORAGE) to the Python import path of your custom storage class.

#### HTTPCache middleware settings

[`HttpCacheMiddleware`](#scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware) can be
configured through the following settings:

<a id="std-setting-HTTPCACHE_ENABLED"></a>

##### HTTPCACHE_ENABLED

Default: `False`

Whether the HTTP cache will be enabled.

<a id="std-setting-HTTPCACHE_EXPIRATION_SECS"></a>

##### HTTPCACHE_EXPIRATION_SECS

Default: `0`

Expiration time for cached requests, in seconds.

Cached requests older than this time will be re-downloaded. If zero, cached
requests will never expire.

<a id="std-setting-HTTPCACHE_DIR"></a>

##### HTTPCACHE_DIR

Default: `'httpcache'`

The directory to use for storing the (low-level) HTTP cache. If empty, the HTTP
cache will be disabled. If a relative path is given, is taken relative to the
project data dir. For more info see: [Default structure of Scrapy projects](commands.md#topics-project-structure).

<a id="std-setting-HTTPCACHE_IGNORE_HTTP_CODES"></a>

##### HTTPCACHE_IGNORE_HTTP_CODES

Default: `[]`

Don’t cache response with these HTTP codes.

<a id="std-setting-HTTPCACHE_IGNORE_MISSING"></a>

##### HTTPCACHE_IGNORE_MISSING

Default: `False`

If enabled, requests not found in the cache will be ignored instead of downloaded.

<a id="std-setting-HTTPCACHE_IGNORE_SCHEMES"></a>

##### HTTPCACHE_IGNORE_SCHEMES

Default: `['file']`

Don’t cache responses with these URI schemes.

<a id="std-setting-HTTPCACHE_STORAGE"></a>

##### HTTPCACHE_STORAGE

Default: `'scrapy.extensions.httpcache.FilesystemCacheStorage'`

The class which implements the cache storage backend.

<a id="std-setting-HTTPCACHE_DBM_MODULE"></a>

##### HTTPCACHE_DBM_MODULE

Default: `'dbm'`

The database module to use in the [DBM storage backend](#httpcache-storage-dbm). This setting is specific to the DBM backend.

<a id="std-setting-HTTPCACHE_POLICY"></a>

##### HTTPCACHE_POLICY

Default: `'scrapy.extensions.httpcache.DummyPolicy'`

The class which implements the cache policy.

<a id="std-setting-HTTPCACHE_GZIP"></a>

##### HTTPCACHE_GZIP

Default: `False`

If enabled, will compress all cached data with gzip.
This setting is specific to the Filesystem backend.

<a id="std-setting-HTTPCACHE_ALWAYS_STORE"></a>

##### HTTPCACHE_ALWAYS_STORE

Default: `False`

If enabled, will cache pages unconditionally.

A spider may wish to have all responses available in the cache, for
future use with `Cache-Control: max-stale`, for instance. The
DummyPolicy caches all responses but never revalidates them, and
sometimes a more nuanced policy is desirable.

This setting still respects `Cache-Control: no-store` directives in responses.
If you don’t want that, filter `no-store` out of the Cache-Control headers in
responses you feed to the cache middleware.

<a id="std-setting-HTTPCACHE_IGNORE_RESPONSE_CACHE_CONTROLS"></a>

##### HTTPCACHE_IGNORE_RESPONSE_CACHE_CONTROLS

Default: `[]`

List of Cache-Control directives in responses to be ignored.

Sites often set “no-store”, “no-cache”, “must-revalidate”, etc., but get
upset at the traffic a spider can generate if it actually respects those
directives. This allows to selectively ignore Cache-Control directives
that are known to be unimportant for the sites being crawled.

We assume that the spider will not issue Cache-Control directives
in requests unless it actually needs them, so directives in requests are
not filtered.

<a id="http-compression"></a>

### HttpCompressionMiddleware

### *class* scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware

This middleware allows compressed (gzip, deflate, [brotli](https://www.ietf.org/rfc/rfc7932.txt), [zstd](https://www.ietf.org/rfc/rfc8478.txt))
traffic to be sent/received from web sites.

#### HttpCompressionMiddleware Settings

<a id="std-setting-COMPRESSION_ENABLED"></a>

##### COMPRESSION_ENABLED

Default: `True`

Whether the Compression middleware will be enabled.

### HttpProxyMiddleware

<a id="std-reqmeta-proxy"></a>

### *class* scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware

This middleware sets the HTTP proxy to use for requests, by setting the
[`proxy`](#std-reqmeta-proxy) meta value for [`Request`](request-response.md#scrapy.Request) objects.

Like the Python standard library module [`urllib.request`](https://docs.python.org/3/library/urllib.request.html#module-urllib.request), it obeys
the following environment variables:

* `http_proxy`
* `https_proxy`
* `no_proxy`

You can also set the meta key [`proxy`](#std-reqmeta-proxy) per-request, to a value like
`http://some_proxy_server:port` or `http://username:password@some_proxy_server:port`.
Keep in mind this value will take precedence over `http_proxy`/`https_proxy`
environment variables, and it will also ignore `no_proxy` environment variable.

> [!NOTE]
> Handling of this meta key needs to be implemented inside the [download
> handler](download-handlers.md#topics-download-handlers), so it’s not guaranteed to be supported
> by all 3rd-party handlers. It’s currently unsupported by
> [`H2DownloadHandler`](download-handlers.md#scrapy.core.downloader.handlers.http2.H2DownloadHandler).

> [!NOTE]
> Usually a proxy URL uses the `http://` scheme. More rarely, it uses the
> `https://` one. While both kinds of proxy URLs can be used with both HTTP
> and HTTPS destination URLs, the specifics of the network exchange are
> different for all 4 cases and it’s possible that HTTPS proxies are fully or
> partially unsupported by a given download handler. Currently,
> [`HTTP11DownloadHandler`](download-handlers.md#scrapy.core.downloader.handlers.http11.HTTP11DownloadHandler)
> supports HTTPS proxies only for HTTP destinations.

> [!NOTE]
> If the download handler supports it, you can use a SOCKS proxy URL (e.g.
> `socks5://username:password@some_proxy_server:port`).
> [`HttpxDownloadHandler`](download-handlers.md#scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler)
> supports SOCKS proxies while other built-in handlers don’t.

#### HttpProxyMiddleware settings

<a id="std-setting-HTTPPROXY_ENABLED"></a>

##### HTTPPROXY_ENABLED

Default: `True`

Whether or not to enable the [`HttpProxyMiddleware`](#scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware).

<a id="std-setting-HTTPPROXY_AUTH_ENCODING"></a>

##### HTTPPROXY_AUTH_ENCODING

Default: `"latin-1"`

The default encoding for proxy authentication on [`HttpProxyMiddleware`](#scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware).

### OffsiteMiddleware

### *class* scrapy.downloadermiddlewares.offsite.OffsiteMiddleware(stats: [StatsCollector](stats.md#scrapy.statscollectors.StatsCollector))

Filter out requests for URLs outside the domains covered by the spider.

> [!WARNING]
> Added in version 2.11.2.

A request is allowed if its host name is in the
[`allowed_domains`](spiders.md#scrapy.Spider.allowed_domains) attribute of the spider, or is a
subdomain of one of those domains. E.g. `www.example.org` also allows
`bob.www.example.org`, but neither `www2.example.org` nor
`example.org`. See [`should_follow()`](#scrapy.downloadermiddlewares.offsite.OffsiteMiddleware.should_follow) to use a different policy.

If the spider does not define [`allowed_domains`](spiders.md#scrapy.Spider.allowed_domains), or
the attribute is empty, every request is allowed.

Filtered requests are logged as follows:

```
DEBUG: Filtered offsite request to 'offsite.example': <GET http://offsite.example/some/page.html>
```

Only the first request filtered for a given domain is logged, to keep the
log readable.

<a id="std-reqmeta-allow_offsite"></a>

#### allow_offsite

Requests with the `allow_offsite` [`meta`](request-response.md#scrapy.Request.meta) key set to
`True`, or with [`dont_filter`](request-response.md#scrapy.Request.dont_filter) set to `True`, are
allowed regardless of their host name.

#### should_follow(request: [Request](request-response.md#scrapy.Request), spider: [Spider](spiders.md#scrapy.spiders.Spider)) → [bool](https://docs.python.org/3/library/functions.html#bool)

Return `True` if *request* is on site, `False` if it must be
filtered out.

Override this method to implement a different offsite policy. For
example, to allow the domains in
[`allowed_domains`](spiders.md#scrapy.Spider.allowed_domains) but none of their subdomains:

```python
from scrapy.downloadermiddlewares.offsite import OffsiteMiddleware
from scrapy.utils.httpobj import urlparse_cached


class RootOnlyOffsiteMiddleware(OffsiteMiddleware):
    def should_follow(self, request, spider):
        return urlparse_cached(request).hostname in spider.allowed_domains
```

### RedirectMiddleware

### *class* scrapy.downloadermiddlewares.redirect.RedirectMiddleware

This middleware handles redirection of requests based on response status.

<a id="std-reqmeta-redirect_urls"></a>

The urls which the request goes through (while being redirected) can be found
in the `redirect_urls` [`Request.meta`](request-response.md#scrapy.Request.meta) key.

<a id="std-reqmeta-redirect_reasons"></a>

The reason behind each redirect in [`redirect_urls`](#std-reqmeta-redirect_urls) can be found in the
`redirect_reasons` [`Request.meta`](request-response.md#scrapy.Request.meta) key. For
example: `[301, 302, 307, 'meta refresh']`.

The format of a reason depends on the middleware that handled the corresponding
redirect. For example, [`RedirectMiddleware`](#scrapy.downloadermiddlewares.redirect.RedirectMiddleware) indicates the triggering
response status code as an integer, while [`MetaRefreshMiddleware`](#scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware)
always uses the `'meta refresh'` string as reason.

<a id="std-reqmeta-redirect_times"></a>

The number of redirects that a request has gone through so far can be found in
the `redirect_times` [`Request.meta`](request-response.md#scrapy.Request.meta) key.

<a id="std-reqmeta-redirect_ttl"></a>

The number of redirects that a request may still go through is tracked in the
`redirect_ttl` [`Request.meta`](request-response.md#scrapy.Request.meta) key, which is
initialized from [`REDIRECT_MAX_TIMES`](#std-setting-REDIRECT_MAX_TIMES) and decreased on every redirect.
Set it on a request to allow fewer redirects for that request;
[`REDIRECT_MAX_TIMES`](#std-setting-REDIRECT_MAX_TIMES) still applies as an upper bound.

The [`RedirectMiddleware`](#scrapy.downloadermiddlewares.redirect.RedirectMiddleware) can be configured through the following
settings (see the settings documentation for more info):

* [`REDIRECT_ENABLED`](#std-setting-REDIRECT_ENABLED)
* [`REDIRECT_MAX_TIMES`](#std-setting-REDIRECT_MAX_TIMES)

<a id="std-reqmeta-dont_redirect"></a>

If [`Request.meta`](request-response.md#scrapy.Request.meta) has `dont_redirect`
key set to True, the request will be ignored by this middleware.

If you want to handle some redirect status codes in your spider, you can
specify these in the `handle_httpstatus_list` spider attribute.

For example, if you want the redirect middleware to ignore 301 and 302
responses (and pass them through to your spider) you can do this:

```python
class MySpider(CrawlSpider):
    handle_httpstatus_list = [301, 302]
```

The `handle_httpstatus_list` key of [`Request.meta`](request-response.md#scrapy.Request.meta) can also be used to specify which response codes to
allow on a per-request basis. You can also set the meta key
`handle_httpstatus_all` to `True` if you want to allow any response code
for a request.

To decide whether to follow a redirect based on its target URL, subclass
[`RedirectMiddleware`](#scrapy.downloadermiddlewares.redirect.RedirectMiddleware) and override
[`process_response()`](#scrapy.downloadermiddlewares.DownloaderMiddleware.process_response):

```python
import re

from scrapy import Request
from scrapy.downloadermiddlewares.redirect import RedirectMiddleware
from scrapy.exceptions import IgnoreRequest


class AllowedRedirectMiddleware(RedirectMiddleware):
    def process_response(self, request, response, spider):
        result = super().process_response(request, response, spider)
        if isinstance(result, Request) and not re.search(r"/product/", result.url):
            raise IgnoreRequest(f"Redirect target not allowed: {result.url}")
        return result
```

Enable it in place of [`RedirectMiddleware`](#scrapy.downloadermiddlewares.redirect.RedirectMiddleware), using the same priority:

```python
from scrapy import Spider


class MySpider(Spider):
    custom_settings = {
        "DOWNLOADER_MIDDLEWARES": {
            "scrapy.downloadermiddlewares.redirect.RedirectMiddleware": None,
            "myproject.middlewares.AllowedRedirectMiddleware": 600,
        }
    }
```

#### RedirectMiddleware settings

<a id="std-setting-REDIRECT_ENABLED"></a>

##### REDIRECT_ENABLED

Default: `True`

Whether the Redirect middleware will be enabled.

<a id="std-setting-REDIRECT_MAX_TIMES"></a>

##### REDIRECT_MAX_TIMES

Default: `20`

The maximum number of redirections that will be followed for a single request.
If maximum redirections are exceeded, the request is aborted and ignored.

### MetaRefreshMiddleware

### *class* scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware

This middleware handles redirection of requests based on meta-refresh html tag.

The [`MetaRefreshMiddleware`](#scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware) can be configured through the following
settings (see the settings documentation for more info):

* [`METAREFRESH_ENABLED`](#std-setting-METAREFRESH_ENABLED)
* [`METAREFRESH_IGNORE_TAGS`](#std-setting-METAREFRESH_IGNORE_TAGS)
* [`METAREFRESH_MAXDELAY`](#std-setting-METAREFRESH_MAXDELAY)

This middleware obey [`REDIRECT_MAX_TIMES`](#std-setting-REDIRECT_MAX_TIMES) setting, [`dont_redirect`](#std-reqmeta-dont_redirect),
[`redirect_urls`](#std-reqmeta-redirect_urls) and [`redirect_reasons`](#std-reqmeta-redirect_reasons) request meta keys as described
for [`RedirectMiddleware`](#scrapy.downloadermiddlewares.redirect.RedirectMiddleware)

#### MetaRefreshMiddleware settings

<a id="std-setting-METAREFRESH_ENABLED"></a>

##### METAREFRESH_ENABLED

Default: `True`

Whether the Meta Refresh middleware will be enabled.

<a id="std-setting-METAREFRESH_IGNORE_TAGS"></a>

##### METAREFRESH_IGNORE_TAGS

Default: `["noscript"]`

Meta tags within these tags are ignored.

> [!WARNING]
> Changed in version 2.11.2: The default value of [`METAREFRESH_IGNORE_TAGS`](#std-setting-METAREFRESH_IGNORE_TAGS) changed from
> `[]` to `["noscript"]`.

<a id="std-setting-METAREFRESH_MAXDELAY"></a>

##### METAREFRESH_MAXDELAY

Default: `100`

The maximum meta-refresh delay (in seconds) to follow the redirection.
Some sites use meta-refresh for redirecting to a session expired page, so we
restrict automatic redirection to the maximum delay.

### RetryMiddleware

### *class* scrapy.downloadermiddlewares.retry.RetryMiddleware

A middleware to retry failed requests that are potentially caused by
temporary problems such as a connection timeout or HTTP 500 error.

<a id="std-reqmeta-dont_retry"></a>

If [`Request.meta`](request-response.md#scrapy.Request.meta) has `dont_retry` key
set to True, the request will be ignored by this middleware.

<a id="std-reqmeta-retry_times"></a>

The number of times that a request has been retried so far can be found in the
`retry_times` [`Request.meta`](request-response.md#scrapy.Request.meta) key. Since it counts
against [`RETRY_TIMES`](#std-setting-RETRY_TIMES), copying it into a follow-up request lowers the
number of retries allowed for that request.

<a id="std-reqmeta-priority_adjust"></a>

The `priority_adjust` [`Request.meta`](request-response.md#scrapy.Request.meta) key
overrides [`RETRY_PRIORITY_ADJUST`](#std-setting-RETRY_PRIORITY_ADJUST) for a request.

To retry requests from a spider callback, you can use the
[`get_retry_request()`](#scrapy.downloadermiddlewares.retry.get_retry_request) function:

### scrapy.downloadermiddlewares.retry.get_retry_request(request: [Request](request-response.md#scrapy.Request), \*, spider: [scrapy.Spider](spiders.md#scrapy.Spider), reason: [str](https://docs.python.org/3/library/stdtypes.html#str) | [Exception](https://docs.python.org/3/library/exceptions.html#Exception) | [type](https://docs.python.org/3/library/functions.html#type)[[Exception](https://docs.python.org/3/library/exceptions.html#Exception)] = 'unspecified', max_retry_times: [int](https://docs.python.org/3/library/functions.html#int) | [None](https://docs.python.org/3/library/constants.html#None) = None, priority_adjust: [int](https://docs.python.org/3/library/functions.html#int) | [None](https://docs.python.org/3/library/constants.html#None) = None, logger: Logger = <Logger scrapy.downloadermiddlewares.retry (WARNING)>, give_up_log_level: [int](https://docs.python.org/3/library/functions.html#int) | [str](https://docs.python.org/3/library/stdtypes.html#str) | [None](https://docs.python.org/3/library/constants.html#None) = None, stats_base_key: [str](https://docs.python.org/3/library/stdtypes.html#str) = 'retry') → [Request](request-response.md#scrapy.Request) | [None](https://docs.python.org/3/library/constants.html#None)

Returns a new [`Request`](request-response.md#scrapy.Request) object to retry the specified
request, or `None` if retries of the specified request have been
exhausted.

For example, in a [`Spider`](spiders.md#scrapy.Spider) callback, you could use it as
follows:

```python
def parse(self, response):
    if not response.text:
        new_request_or_none = get_retry_request(
            response.request,
            spider=self,
            reason="empty",
        )
        return new_request_or_none
```

*spider* is the [`Spider`](spiders.md#scrapy.Spider) instance which is asking for the
retry request. It is used to access the [settings](settings.md#topics-settings)
and [stats](stats.md#topics-stats), and to provide extra logging context (see
[`logging.debug()`](https://docs.python.org/3/library/logging.html#logging.debug)).

*reason* is a string, an [`Exception`](https://docs.python.org/3/library/exceptions.html#Exception) subclass or an
[`Exception`](https://docs.python.org/3/library/exceptions.html#Exception) object that indicates the reason why the request needs
to be retried. It is used to name retry stats.

*max_retry_times* is a number that determines the maximum number of times
that *request* can be retried. If not specified or `None`, the number is
read from the [`max_retry_times`](request-response.md#std-reqmeta-max_retry_times) meta key of the request. If the
[`max_retry_times`](request-response.md#std-reqmeta-max_retry_times) meta key is not defined or `None`, the number
is read from the [`RETRY_TIMES`](#std-setting-RETRY_TIMES) setting.

*priority_adjust* is a number that determines how the priority of the new
request changes in relation to *request*. If not specified, the number is
read from the [`RETRY_PRIORITY_ADJUST`](#std-setting-RETRY_PRIORITY_ADJUST) setting.

*logger* is the logging.Logger object to be used when logging messages

*give_up_log_level* is the [logging level](https://docs.python.org/3/library/logging.html#levels) used for the
message logged when a request exceeds its retries. See
[`RETRY_GIVE_UP_LOG_LEVEL`](#std-setting-RETRY_GIVE_UP_LOG_LEVEL) for details.

> [!WARNING]
> Added in version 2.17.0: The *give_up_log_level* parameter.

*stats_base_key* is a string to be used as the base key for the
retry-related job stats

#### RetryMiddleware Settings

<a id="std-setting-RETRY_ENABLED"></a>

##### RETRY_ENABLED

Default: `True`

Whether the Retry middleware will be enabled.

<a id="std-setting-RETRY_TIMES"></a>

##### RETRY_TIMES

Default: `2`

Maximum number of times to retry, in addition to the first download.

Maximum number of retries can also be specified per-request using
[`max_retry_times`](request-response.md#std-reqmeta-max_retry_times) attribute of [`Request.meta`](request-response.md#scrapy.Request.meta).
When initialized, the [`max_retry_times`](request-response.md#std-reqmeta-max_retry_times) meta key takes higher
precedence over the [`RETRY_TIMES`](#std-setting-RETRY_TIMES) setting.

<a id="std-setting-RETRY_HTTP_CODES"></a>

##### RETRY_HTTP_CODES

Default: `[500, 502, 503, 504, 522, 524, 408, 429]`

Which HTTP response codes to retry. Other errors (DNS lookup issues,
connections lost, etc) are always retried.

In some cases you may want to add 400 to [`RETRY_HTTP_CODES`](#std-setting-RETRY_HTTP_CODES) because
it is a common code used to indicate server overload. It is not included by
default because HTTP specs say so.

<a id="std-setting-RETRY_EXCEPTIONS"></a>

##### RETRY_EXCEPTIONS

Default:

```
[
    'scrapy.exceptions.CannotResolveHostError',
    'scrapy.exceptions.DownloadConnectionRefusedError',
    'scrapy.exceptions.DownloadFailedError',
    'scrapy.exceptions.DownloadTimeoutError',
    'scrapy.exceptions.ResponseDataLossError',
    'twisted.internet.error.ConnectionDone',
    'twisted.internet.error.ConnectError',
    'twisted.internet.error.ConnectionLost',
    OSError,
    'scrapy.core.downloader.handlers.http11.TunnelError',
]
```

List of exceptions to retry.

Each list entry may be an exception type or its import path as a string.

An exception will not be caught when the exception type is not in
[`RETRY_EXCEPTIONS`](#std-setting-RETRY_EXCEPTIONS) or when the maximum number of retries for a request
has been exceeded (see [`RETRY_TIMES`](#std-setting-RETRY_TIMES)). To learn about uncaught
exception propagation, see
[`process_exception()`](#scrapy.downloadermiddlewares.DownloaderMiddleware.process_exception).

<a id="std-setting-RETRY_GIVE_UP_LOG_LEVEL"></a>

##### RETRY_GIVE_UP_LOG_LEVEL

> [!WARNING]
> Added in version 2.17.0.

Default: `"ERROR"`

[Logging level](https://docs.python.org/3/library/logging.html#levels) used for the message logged when a request
exceeds its retries.

Can be a level name (e.g. `"WARNING"`) or a number (e.g. `logging.WARNING`
or `30`).

See also: [`give_up_log_level`](request-response.md#std-reqmeta-give_up_log_level), [`get_retry_request()`](#scrapy.downloadermiddlewares.retry.get_retry_request).

<a id="std-setting-RETRY_PRIORITY_ADJUST"></a>

##### RETRY_PRIORITY_ADJUST

Default: `-1`

Adjust retry request priority relative to original request:

- a positive priority adjust means higher priority.
- **a negative priority adjust (default) means lower priority.**

See also: [`priority_adjust`](#std-reqmeta-priority_adjust).

<a id="topics-dlmw-robots"></a>

### RobotsTxtMiddleware

### *class* scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware

This middleware filters out requests forbidden by the robots.txt exclusion
standard.

To make sure Scrapy respects robots.txt make sure the middleware is enabled
and the [`ROBOTSTXT_OBEY`](settings.md#std-setting-ROBOTSTXT_OBEY) setting is enabled.

The [`ROBOTSTXT_USER_AGENT`](settings.md#std-setting-ROBOTSTXT_USER_AGENT) setting can be used to specify the
user agent string to use for matching in the [robots.txt](https://www.robotstxt.org/) file. If it
is `None`, the User-Agent header you are sending with the request or the
[`USER_AGENT`](settings.md#std-setting-USER_AGENT) setting (in that order) will be used for determining
the user agent to use in the [robots.txt](https://www.robotstxt.org/) file.

This middleware has to be combined with a [robots.txt](https://www.robotstxt.org/) parser.

Scrapy ships with support for the following [robots.txt](https://www.robotstxt.org/) parsers:

* [Protego](#protego-parser) (default)
* [RobotFileParser](#python-robotfileparser)
* [Robotexclusionrulesparser](#rerp-parser)

You can change the [robots.txt](https://www.robotstxt.org/) parser with the [`ROBOTSTXT_PARSER`](settings.md#std-setting-ROBOTSTXT_PARSER)
setting. Or you can also [implement support for a new parser](#support-for-new-robots-parser).

<a id="std-reqmeta-dont_obey_robotstxt"></a>

If [`Request.meta`](request-response.md#scrapy.Request.meta) has
`dont_obey_robotstxt` key set to True
the request will be ignored by this middleware even if
[`ROBOTSTXT_OBEY`](settings.md#std-setting-ROBOTSTXT_OBEY) is enabled.

Parsers vary in several aspects:

* Language of implementation
* Supported specification
* Support for wildcard matching
* Usage of [length based rule](https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec#order-of-precedence-for-rules):
  in particular for `Allow` and `Disallow` directives, where the most
  specific rule based on the length of the path trumps the less specific
  (shorter) rule

Performance comparison of different parsers is available at [the following link](https://github.com/scrapy/scrapy/issues/3969).

<a id="protego-parser"></a>

#### Protego parser

Based on [Protego](https://github.com/scrapy/protego):

* implemented in Python
* is compliant with [Google’s Robots.txt Specification](https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec)
* supports wildcard matching
* uses the length based rule

Scrapy uses this parser by default.

<a id="python-robotfileparser"></a>

#### RobotFileParser

Based on [`RobotFileParser`](https://docs.python.org/3/library/urllib.robotparser.html#urllib.robotparser.RobotFileParser):

* is Python’s built-in [robots.txt](https://www.robotstxt.org/) parser
* is compliant with [Martijn Koster’s 1996 draft specification](https://www.robotstxt.org/norobots-rfc.txt)
* lacks support for wildcard matching (before Python 3.14.5)
* doesn’t use the length based rule (before Python 3.14.5)

It is faster than Protego and backward-compatible with versions of Scrapy before 1.8.0.

In order to use this parser, set:

* [`ROBOTSTXT_PARSER`](settings.md#std-setting-ROBOTSTXT_PARSER) to `scrapy.robotstxt.PythonRobotParser`

<a id="rerp-parser"></a>

#### Robotexclusionrulesparser

Based on [Robotexclusionrulesparser](https://pypi.org/project/robotexclusionrulesparser/):

* implemented in Python
* is compliant with [Martijn Koster’s 1996 draft specification](https://www.robotstxt.org/norobots-rfc.txt)
* supports wildcard matching
* doesn’t use the length based rule

In order to use this parser:

* Install the [robotparser](../intro/install.md#extras) extra.
* Set [`ROBOTSTXT_PARSER`](settings.md#std-setting-ROBOTSTXT_PARSER) setting to
  `scrapy.robotstxt.RerpRobotParser`

<a id="support-for-new-robots-parser"></a>

#### Implementing support for a new parser

You can implement support for a new [robots.txt](https://www.robotstxt.org/) parser by subclassing
the abstract base class [`RobotParser`](#scrapy.robotstxt.RobotParser) and
implementing the methods described below.

<a id="module-scrapy.robotstxt"></a>

### *class* scrapy.robotstxt.RobotParser

#### *abstractmethod classmethod* from_crawler(crawler: [Crawler](api.md#scrapy.crawler.Crawler), robotstxt_body: [bytes](https://docs.python.org/3/library/stdtypes.html#bytes)) → Self

Parse the content of a [robots.txt](https://www.robotstxt.org/) file as bytes. This must be a class method.
It must return a new instance of the parser backend.

* **Parameters:**
  * **crawler** ([`Crawler`](api.md#scrapy.crawler.Crawler) instance) – crawler which made the request
  * **robotstxt_body** ([*bytes*](https://docs.python.org/3/library/stdtypes.html#bytes)) – content of a [robots.txt](https://www.robotstxt.org/) file.

#### *abstractmethod* allowed(url: [str](https://docs.python.org/3/library/stdtypes.html#str) | [bytes](https://docs.python.org/3/library/stdtypes.html#bytes), user_agent: [str](https://docs.python.org/3/library/stdtypes.html#str) | [bytes](https://docs.python.org/3/library/stdtypes.html#bytes)) → [bool](https://docs.python.org/3/library/functions.html#bool)

Return `True` if  `user_agent` is allowed to crawl `url`, otherwise return `False`.

* **Parameters:**
  * **url** ([*str*](https://docs.python.org/3/library/stdtypes.html#str) *or* [*bytes*](https://docs.python.org/3/library/stdtypes.html#bytes)) – Absolute URL
  * **user_agent** ([*str*](https://docs.python.org/3/library/stdtypes.html#str) *or* [*bytes*](https://docs.python.org/3/library/stdtypes.html#bytes)) – User agent

#### crawl_delay(user_agent: [str](https://docs.python.org/3/library/stdtypes.html#str) | [bytes](https://docs.python.org/3/library/stdtypes.html#bytes)) → [float](https://docs.python.org/3/library/functions.html#float) | [None](https://docs.python.org/3/library/constants.html#None)

Return the `Crawl-delay` directive for `user_agent` as a number
of seconds, or `None` if it is not set or the backend does not support
it.

> [!WARNING]
> Added in version 2.18.0.

### DownloaderStats

### *class* scrapy.downloadermiddlewares.stats.DownloaderStats

Middleware that stores stats of all requests, responses and exceptions that
pass through it.

To use this middleware you must enable the [`DOWNLOADER_STATS`](settings.md#std-setting-DOWNLOADER_STATS)
setting.

### UserAgentMiddleware

### *class* scrapy.downloadermiddlewares.useragent.UserAgentMiddleware

Middleware that sets the `User-Agent` header.

The header value is taken from the [`USER_AGENT`](settings.md#std-setting-USER_AGENT) setting.