Exceptions

Here’s a list of all exceptions included in Scrapy and their usage, except for the download handler exceptions.

exception scrapy.exceptions.CloseSpider(reason: str = 'cancelled')

Raised from a spider callback to request the spider to be closed/stopped.

reason is a string with the reason for closing.

For example:

def parse_page(self, response):
    if "Bandwidth exceeded" in response.text:
        raise CloseSpider("bandwidth_exceeded")

exception scrapy.exceptions.DontCloseSpider

Raised in a spider_idle signal handler to prevent the spider from being closed.

exception scrapy.exceptions.DropItem(message: str, log_level: str | None = None)

Raised from the process_item() method of an item pipeline to stop the processing of an item.

exception scrapy.exceptions.IgnoreRequest

Raised to indicate that a request should be ignored.

A downloader middleware can raise it from its process_request() or process_response() method to drop a request, and a request_scheduled signal handler can raise it to drop a request before it reaches the scheduler.

exception scrapy.exceptions.NotConfigured

Raised by a component from its __init__() or from_crawler() method to indicate that it will remain disabled.

Only the following components can be disabled this way:

• Downloader middlewares

• Extensions

• Item pipelines

• Spider middlewares

exception scrapy.exceptions.NotSupported

Raised to indicate that a requested feature is not supported.

For example, Scrapy raises it when text-parsing shortcuts such as response.css() or response.xpath() are used on a Response whose content is not text, or when sending a request whose URL scheme has no matching download handler.

exception scrapy.exceptions.StopDownload(*, fail: bool = True)

Raised from a bytes_received or headers_received signal handler to stop the download of the response body.

The fail boolean parameter controls which method will handle the resulting response:

• If fail=True (default), the request errback is called. The response object is available as the response attribute of the StopDownload exception, which is in turn stored as the value attribute of the received Failure object. This means that in an errback defined as def errback(self, failure), the response can be accessed though failure.value.response.

• If fail=False, the request callback is called instead.

In both cases, the response could have its body truncated: the body contains all bytes received up until the exception is raised, including the bytes received in the signal handler that raises the exception. Also, the response object is marked with "download_stopped" in its flags attribute.