Extensions
Extensions are components that allow inserting your own custom functionality into Scrapy.
Unlike other components, extensions do not have a specific role in Scrapy. They are “wildcard” components that can be used for anything that does not fit the role of any other type of component.
Loading and activating extensions
Extensions are loaded at startup by creating a single instance of the extension class per spider being run.
To enable an extension, add it to the EXTENSIONS
setting. For
example:
EXTENSIONS = {
"scrapy.extensions.corestats.CoreStats": 500,
"scrapy.extensions.telnet.TelnetConsole": 500,
}
EXTENSIONS
is merged with EXTENSIONS_BASE
(not meant to
be overridden), and the priorities in the resulting value determine the
loading order.
As extensions typically do not depend on each other, their loading order is
irrelevant in most cases. This is why the EXTENSIONS_BASE
setting
defines all extensions with the same order (0
). However, you may need to
carefully use priorities if you add an extension that depends on other
extensions being already loaded.
Writing your own extension
Each extension is a component.
Typically, extensions connect to signals and perform tasks triggered by them.
Sample extension
Here we will implement a simple extension to illustrate the concepts described in the previous section. This extension will log a message every time:
a spider is opened
a spider is closed
a specific number of items are scraped
The extension will be enabled through the MYEXT_ENABLED
setting and the
number of items will be specified through the MYEXT_ITEMCOUNT
setting.
Here is the code of such extension:
import logging
from scrapy import signals
from scrapy.exceptions import NotConfigured
logger = logging.getLogger(__name__)
class SpiderOpenCloseLogging:
def __init__(self, item_count):
self.item_count = item_count
self.items_scraped = 0
@classmethod
def from_crawler(cls, crawler):
# first check if the extension should be enabled and raise
# NotConfigured otherwise
if not crawler.settings.getbool("MYEXT_ENABLED"):
raise NotConfigured
# get the number of items from settings
item_count = crawler.settings.getint("MYEXT_ITEMCOUNT", 1000)
# instantiate the extension object
ext = cls(item_count)
# connect the extension object to signals
crawler.signals.connect(ext.spider_opened, signal=signals.spider_opened)
crawler.signals.connect(ext.spider_closed, signal=signals.spider_closed)
crawler.signals.connect(ext.item_scraped, signal=signals.item_scraped)
# return the extension object
return ext
def spider_opened(self, spider):
logger.info("opened spider %s", spider.name)
def spider_closed(self, spider):
logger.info("closed spider %s", spider.name)
def item_scraped(self, item, spider):
self.items_scraped += 1
if self.items_scraped % self.item_count == 0:
logger.info("scraped %d items", self.items_scraped)
Built-in extensions reference
General purpose extensions
Log Stats extension
Log basic stats like crawled pages and scraped items.
Core Stats extension
Enable the collection of core statistics, provided the stats collection is enabled (see Stats Collection).
Telnet console extension
Provides a telnet console for getting into a Python interpreter inside the currently running Scrapy process, which can be very useful for debugging.
The telnet console must be enabled by the TELNETCONSOLE_ENABLED
setting, and the server will listen in the port specified in
TELNETCONSOLE_PORT
.
Memory usage extension
Note
This extension does not work in Windows.
Monitors the memory used by the Scrapy process that runs the spider and:
sends a notification e-mail when it exceeds a certain value
closes the spider when it exceeds a certain value
The notification e-mails can be triggered when a certain warning value is
reached (MEMUSAGE_WARNING_MB
) and when the maximum value is reached
(MEMUSAGE_LIMIT_MB
) which will also cause the spider to be closed
and the Scrapy process to be terminated.
This extension is enabled by the MEMUSAGE_ENABLED
setting and
can be configured with the following settings:
Memory debugger extension
An extension for debugging memory usage. It collects information about:
objects uncollected by the Python garbage collector
objects left alive that shouldn’t. For more info, see Debugging memory leaks with trackref
To enable this extension, turn on the MEMDEBUG_ENABLED
setting. The
info will be stored in the stats.
Spider state extension
Manages spider state data by loading it before a crawl and saving it after.
Give a value to the JOBDIR
setting to enable this extension.
When enabled, this extension manages the state
attribute of your Spider
instance:
When your spider closes (
spider_closed
), the contents of itsstate
attribute are serialized into a file namedspider.state
in theJOBDIR
folder.When your spider opens (
spider_opened
), if a previously-generatedspider.state
file exists in theJOBDIR
folder, it is loaded into thestate
attribute.
For an example, see Keeping persistent state between batches.
Close spider extension
Closes a spider automatically when some conditions are met, using a specific closing reason for each condition.
The conditions for closing a spider can be configured through the following settings:
Note
When a certain closing condition is met, requests which are
currently in the downloader queue (up to CONCURRENT_REQUESTS
requests) are still processed.
CLOSESPIDER_TIMEOUT
Default: 0
An integer which specifies a number of seconds. If the spider remains open for
more than that number of second, it will be automatically closed with the
reason closespider_timeout
. If zero (or non set), spiders won’t be closed by
timeout.
CLOSESPIDER_TIMEOUT_NO_ITEM
Default: 0
An integer which specifies a number of seconds. If the spider has not produced
any items in the last number of seconds, it will be closed with the reason
closespider_timeout_no_item
. If zero (or non set), spiders won’t be closed
regardless if it hasn’t produced any items.
CLOSESPIDER_ITEMCOUNT
Default: 0
An integer which specifies a number of items. If the spider scrapes more than
that amount and those items are passed by the item pipeline, the
spider will be closed with the reason closespider_itemcount
.
If zero (or non set), spiders won’t be closed by number of passed items.
CLOSESPIDER_PAGECOUNT
Default: 0
An integer which specifies the maximum number of responses to crawl. If the spider
crawls more than that, the spider will be closed with the reason
closespider_pagecount
. If zero (or non set), spiders won’t be closed by
number of crawled responses.
CLOSESPIDER_PAGECOUNT_NO_ITEM
Default: 0
An integer which specifies the maximum number of consecutive responses to crawl
without items scraped. If the spider crawls more consecutive responses than that
and no items are scraped in the meantime, the spider will be closed with the
reason closespider_pagecount_no_item
. If zero (or not set), spiders won’t be
closed by number of crawled responses with no items.
CLOSESPIDER_ERRORCOUNT
Default: 0
An integer which specifies the maximum number of errors to receive before
closing the spider. If the spider generates more than that number of errors,
it will be closed with the reason closespider_errorcount
. If zero (or non
set), spiders won’t be closed by number of errors.
StatsMailer extension
This simple extension can be used to send a notification e-mail every time a
domain has finished scraping, including the Scrapy stats collected. The email
will be sent to all recipients specified in the STATSMAILER_RCPTS
setting.
Emails can be sent using the MailSender
class. To see a
full list of parameters, including examples on how to instantiate
MailSender
and use mail settings, see
Sending e-mail.
Periodic log extension
This extension periodically logs rich stat data as a JSON object:
2023-08-04 02:30:57 [scrapy.extensions.logstats] INFO: Crawled 976 pages (at 162 pages/min), scraped 925 items (at 161 items/min)
2023-08-04 02:30:57 [scrapy.extensions.periodic_log] INFO: {
"delta": {
"downloader/request_bytes": 55582,
"downloader/request_count": 162,
"downloader/request_method_count/GET": 162,
"downloader/response_bytes": 618133,
"downloader/response_count": 162,
"downloader/response_status_count/200": 162,
"item_scraped_count": 161
},
"stats": {
"downloader/request_bytes": 338243,
"downloader/request_count": 992,
"downloader/request_method_count/GET": 992,
"downloader/response_bytes": 3836736,
"downloader/response_count": 976,
"downloader/response_status_count/200": 976,
"item_scraped_count": 925,
"log_count/INFO": 21,
"log_count/WARNING": 1,
"scheduler/dequeued": 992,
"scheduler/dequeued/memory": 992,
"scheduler/enqueued": 1050,
"scheduler/enqueued/memory": 1050
},
"time": {
"elapsed": 360.008903,
"log_interval": 60.0,
"log_interval_real": 60.006694,
"start_time": "2023-08-03 23:24:57",
"utcnow": "2023-08-03 23:30:57"
}
}
This extension logs the following configurable sections:
"delta"
shows how some numeric stats have changed since the last stats log message.The
PERIODIC_LOG_DELTA
setting determines the target stats. They must haveint
orfloat
values."stats"
shows the current value of some stats.The
PERIODIC_LOG_STATS
setting determines the target stats."time"
shows detailed timing data.The
PERIODIC_LOG_TIMING_ENABLED
setting determines whether or not to show this section.
This extension logs data at the start, then on a fixed time interval
configurable through the LOGSTATS_INTERVAL
setting, and finally
right before the crawl ends.
Example extension configuration:
custom_settings = {
"LOG_LEVEL": "INFO",
"PERIODIC_LOG_STATS": {
"include": ["downloader/", "scheduler/", "log_count/", "item_scraped_count/"],
},
"PERIODIC_LOG_DELTA": {"include": ["downloader/"]},
"PERIODIC_LOG_TIMING_ENABLED": True,
"EXTENSIONS": {
"scrapy.extensions.periodic_log.PeriodicLog": 0,
},
}
PERIODIC_LOG_DELTA
Default: None
"PERIODIC_LOG_DELTA": True
- show deltas for allint
andfloat
stat values."PERIODIC_LOG_DELTA": {"include": ["downloader/", "scheduler/"]}
- show deltas for stats with names containing any configured substring."PERIODIC_LOG_DELTA": {"exclude": ["downloader/"]}
- show deltas for all stats with names not containing any configured substring.
PERIODIC_LOG_STATS
Default: None
"PERIODIC_LOG_STATS": True
- show the current value of all stats."PERIODIC_LOG_STATS": {"include": ["downloader/", "scheduler/"]}
- show current values for stats with names containing any configured substring."PERIODIC_LOG_STATS": {"exclude": ["downloader/"]}
- show current values for all stats with names not containing any configured substring.
PERIODIC_LOG_TIMING_ENABLED
Default: False
True
enables logging of timing data (i.e. the "time"
section).
Debugging extensions
Stack trace dump extension
- class scrapy.extensions.periodic_log.StackTraceDump
Dumps information about the running process when a SIGQUIT or SIGUSR2 signal is received. The information dumped is the following:
engine status (using
scrapy.utils.engine.get_engine_status()
)live references (see Debugging memory leaks with trackref)
stack trace of all threads
After the stack trace and engine status is dumped, the Scrapy process continues running normally.
This extension only works on POSIX-compliant platforms (i.e. not Windows), because the SIGQUIT and SIGUSR2 signals are not available on Windows.
There are at least two ways to send Scrapy the SIGQUIT signal:
By pressing Ctrl-while a Scrapy process is running (Linux only?)
By running this command (assuming
<pid>
is the process id of the Scrapy process):kill -QUIT <pid>
Debugger extension
- class scrapy.extensions.periodic_log.Debugger
Invokes a Python debugger inside a running Scrapy process when a SIGUSR2 signal is received. After the debugger is exited, the Scrapy process continues running normally.
This extension only works on POSIX-compliant platforms (i.e. not Windows).