Debugging Spiders

This document explains the most common techniques for debugging spiders. Consider the following Scrapy spider below:

import scrapy
from myproject.items import MyItem


class MySpider(scrapy.Spider):
    name = "myspider"
    start_urls = (
        "http://example.com/page1",
        "http://example.com/page2",
    )

    def parse(self, response):
        # <processing code not shown>
        # collect `item_urls`
        for item_url in item_urls:
            yield scrapy.Request(item_url, self.parse_item)

    def parse_item(self, response):
        # <processing code not shown>
        item = MyItem()
        # populate `item` fields
        # and extract item_details_url
        yield scrapy.Request(
            item_details_url, self.parse_details, cb_kwargs={"item": item}
        )

    def parse_details(self, response, item):
        # populate more `item` fields
        return item

Basically this is a simple spider which parses two pages of items (the start_urls). Items also have a details page with additional information, so we use the cb_kwargs functionality of Request to pass a partially populated item.

Parse Command

The most basic way of checking the output of your spider is to use the parse command. It allows to check the behaviour of different parts of the spider at the method level. It has the advantage of being flexible and simple to use, but does not allow debugging code inside a method.

In order to see the item scraped from a specific url:

$ scrapy parse --spider=myspider -c parse_item -d 2 <item_url>
[ ... scrapy log lines crawling example.com spider ... ]

>>> STATUS DEPTH LEVEL 2 <<<
# Scraped Items  ------------------------------------------------------------
[{'url': <item_url>}]

# Requests  -----------------------------------------------------------------
[]

Using the --verbose or -v option we can see the status at each depth level:

$ scrapy parse --spider=myspider -c parse_item -d 2 -v <item_url>
[ ... scrapy log lines crawling example.com spider ... ]

>>> DEPTH LEVEL: 1 <<<
# Scraped Items  ------------------------------------------------------------
[]

# Requests  -----------------------------------------------------------------
[<GET item_details_url>]


>>> DEPTH LEVEL: 2 <<<
# Scraped Items  ------------------------------------------------------------
[{'url': <item_url>}]

# Requests  -----------------------------------------------------------------
[]

Checking items scraped from a single start_url, can also be easily achieved using:

$ scrapy parse --spider=myspider -d 3 'http://example.com/page1'

Scrapy Shell

While the parse command is very useful for checking behaviour of a spider, it is of little help to check what happens inside a callback, besides showing the response received and the output. How to debug the situation when parse_details sometimes receives no item?

Fortunately, the shell is your bread and butter in this case (see Invoking the shell from spiders to inspect responses):

from scrapy.shell import inspect_response


def parse_details(self, response, item=None):
    if item:
        # populate more `item` fields
        return item
    else:
        inspect_response(response, self)

See also: Invoking the shell from spiders to inspect responses.

Open in browser

Sometimes you just want to see how a certain response looks in a browser, you can use the open_in_browser() function for that:

scrapy.utils.response.open_in_browser(response: TextResponse, _openfunc: Callable[[str], Any] = <function open>) → Any[source]

Open response in a local web browser, adjusting the base tag for external links to work, e.g. so that images and styles are displayed.

For example:

from scrapy.utils.response import open_in_browser


def parse_details(self, response):
    if "item name" not in response.text:
        open_in_browser(response)

On the Windows Subsystem for Linux, set the BROWSER environment variable to wslview to open the response in a Windows browser, which cannot read Linux paths otherwise.

Logging

Logging is another useful option for getting information about your spider run. Although not as convenient, it comes with the advantage that the logs will be available in all future runs should they be necessary again:

def parse_details(self, response, item=None):
    if item:
        # populate more `item` fields
        return item
    else:
        self.logger.warning("No item received for %s", response.url)

For more information, check the Logging section.

Inspecting live traffic

Sometimes it’s important to see what exactly was sent to the server or received from it, such as header values, formatting and order (Scrapy cannot log this, as underlying HTTP libraries produce the final values for request headers and canonicalize response ones) or TLS handshake details. There are two ways to see and log the real traffic of a running spider:

  • Capture the traffic with a tool such as Wireshark. As your requests likely use TLS, you will need to decrypt the traffic (see the Wireshark TLS documentation for detailed instructions). You will need the encryption key which you can save as described in Decrypting TLS traffic. As this way of capturing traffic is passive, it cannot interfere with the spider.

  • Use mitmproxy between the spider and the server, as described below. This is easier to set up and in addition to inspecting the traffic allows modifying it, but it’s not passive: there is now a connection between Scrapy and mitmproxy and another one between mitmproxy and the server instead of a direct connection between Scrapy and the server. Due to this, low-level connection behavior is different from normal crawls, which may change the server behavior, and you cannot easily use mitmproxy and regular proxies in the same crawl.

Using mitmdump

You should refer to the mitmproxy documentation for more details, additional interception modes and advanced features but here is one simple way to use it. First, run a mitmdump instance (it will use the port 8080 by default), asking it to log the traffic details on the terminal (--flow-detail 2 will log headers but not bodies):

mitmdump --flow-detail 2

Then configure http://127.0.0.1:8080 as a proxy in your spider using HttpProxyMiddleware:

https_proxy=http://127.0.0.1:8080 scrapy crawl myspider

To inspect only some requests, set their proxy meta key instead.

Decrypting TLS traffic

Scrapy writes the session keys of its HTTPS connections to the file that the SSLKEYLOGFILE environment variable points to, using the NSS key log format that traffic analysis tools such as Wireshark understand.

Added in version VERSION.

SSLKEYLOGFILE=/tmp/sslkeylog scrapy crawl myspider

Warning

Anyone who can read the key log file can decrypt the traffic of the connections recorded in it, including any credentials that they carry.

Visual Studio Code

To debug spiders with Visual Studio Code you can use the following launch.json:

{
    "version": "0.1.0",
    "configurations": [
        {
            "name": "Python: Launch Scrapy Spider",
            "type": "python",
            "request": "launch",
            "module": "scrapy",
            "args": [
                "runspider",
                "${file}"
            ],
            "console": "integratedTerminal"
        }
    ]
}

Also, make sure you enable “User Uncaught Exceptions”, to catch exceptions in your Scrapy spider.