Telnet Console

Scrapy comes with a built-in telnet console for inspecting and controlling a Scrapy running process. The telnet console is just a regular python shell running inside the Scrapy process, so you can do literally anything from it.

The telnet console is a built-in Scrapy extension which comes enabled by default, but you can also disable it if you want. For more information about the extension itself see Telnet console extension.

How to access the telnet console

The telnet console listens in the TCP port defined in the TELNETCONSOLE_PORT setting, which defaults to 6023. To access the console you need to type:

telnet localhost 6023

You need the telnet program which comes installed by default in Windows, and most Linux distros.

Available variables in the telnet console

The telnet console is like a regular Python shell running inside the Scrapy process, so you can do anything from it including importing new modules, etc.

However, the telnet console comes with some default variables defined for convenience:

Shortcut Description
crawler the Scrapy Crawler object (scrapy.crawler)
engine the Scrapy Engine object (scrapy.core.engine)
spider the spider object (only if there is a single spider opened)
slot the engine slot (only if there is a single spider opened)
extensions the Extension Manager (scrapy.project.crawler.extensions)
stats the Stats Collector (scrapy.stats.stats)
settings the Scrapy settings object (scrapy.conf.settings)
est print a report of the current engine status
prefs for memory debugging (see Debugging memory leaks)
p a shortcut to the pprint.pprint function
hpy for memory debugging (see Debugging memory leaks)

Telnet console usage examples

Here are some example tasks you can do with the telnet console:

View engine status

You can use the est() method of the Scrapy engine to quickly show its state using the telnet console:

telnet localhost 6023
>>> est()
Execution engine status

time()-engine.start_time                        : 21.3188259602
engine.is_idle()                                : False
engine.has_capacity()                           : True
engine.scheduler.is_idle()                      : False
len(engine.scheduler.pending_requests)          : 1
engine.downloader.is_idle()                     : False
len(engine.downloader.slots)                    : 1
engine.scraper.is_idle()                        : False
len(engine.scraper.slots)                       : 1

Spider: <GayotSpider 'gayotcom' at 0x2dc2b10>
  engine.spider_is_idle(spider)                      : False
  engine.slots[spider].closing                       : False
  len(engine.scheduler.pending_requests[spider])     : 11504
  len(engine.downloader.slots[spider].queue)         : 9
  len(engine.downloader.slots[spider].active)        : 17
  len(engine.downloader.slots[spider].transferring)  : 8
  engine.downloader.slots[spider].lastseen           : 1311311093.61
  len(engine.scraper.slots[spider].queue)            : 0
  len(engine.scraper.slots[spider].active)           : 0
  engine.scraper.slots[spider].active_size           : 0
  engine.scraper.slots[spider].itemproc_size         : 0
  engine.scraper.slots[spider].needs_backout()       : False

Pause, resume and stop the Scrapy engine

To pause:

telnet localhost 6023
>>> engine.pause()

To resume:

telnet localhost 6023
>>> engine.unpause()

To stop:

telnet localhost 6023
>>> engine.stop()
Connection closed by foreign host.

Telnet Console signals


Sent just before the telnet console is opened. You can hook up to this signal to add, remove or update the variables that will be available in the telnet local namespace. In order to do that, you need to update the telnet_vars dict in your handler.

Parameters:telnet_vars (dict) – the dict of telnet variables

Telnet settings

These are the settings that control the telnet console’s behaviour:


Default: [6023, 6073]

The port range to use for the etlnet console. If set to None or 0, a dynamically assigned port is used.


Default: ''

The interface the telnet console should listen on