Scrapy is controlled through the control script. The script provides several commands, for different purposes. Each command supports its own particular syntax. In other words, each command supports a different set of arguments and options.

This page doesn’t describe each command and its syntax, but provides an introduction to how the script is used. After you learn how to use it, you can get help for each particular command using the same script.

Global and project-specific

There is one global script shipped with Scrapy and another script automatically created inside your Scrapy project. The project-specific is just a thin wrapper around the global which populates the settings of your project, so you don’t have to specify them every time through the --settings argument.

Using the script

The first thing you would do with the script is create your Scrapy project: startproject myproject

That will create a Scrapy project under the myproject directory and will put a new inside that directory.

So, you go inside the new project directory:

cd myproject

And you’re ready to use your project’s For example, to create a new spider:

python genspider mydomain

This is the same as using the global script and passing the project settings module in the --settings argument: --settings=myproject.settings genspider mydomain

You’ll typically use the project-specific, for convenience.

See all available commands

To see all available commands type: -h

That will print a summary of all available Scrapy commands.

The first line will print the currently active project (if any).

Example (active project):

Scrapy X.X.X - project: myproject



Example (no active project):

Scrapy X.X.X - no active project



Get help for a particular command

To get help about a particular command, including its description, usage and available options type: <command> -h

Example: crawl -h

Using outside your project

Not all commands must be run from “inside” a Scrapy project. You can, for example, use the fetch command to download a page (using Scrapy built-in downloader) from outside a project. Other commands that can be used outside a project are startproject (obviously) and shell, to launch a Scrapy Shell.