This document describes how to install Scrapy on Linux, Windows and Mac OS X systems and it consists of the following 3 steps:
If you use Ubuntu (9.10 or above), you can use the Ubuntu packages to install Scrapy more easily, skipping these steps. The Ubuntu packages are also kept updated with latest bug fixes.
- Python 2.5, 2.6, 2.7 (3.x is not yet supported)
- Twisted 2.5.0, 8.0 or above (Windows users: you’ll need to install Zope.Interface and maybe pywin32 because of this Twisted bug)
- libxml2 (versions prior to 2.6.28 are known to have problems parsing certain malformed HTML, and have also been reported to contain leaks, so 2.6.28 or above is highly recommended)
Step 1. Install Python¶
Scrapy works with Python 2.5, 2.6 or 2.7, which you can get at http://www.python.org/download/
Step 2. Install required libraries¶
The procedure for installing the required third party libraries depends on the platform and operating system you use.
If you’re running Ubuntu/Debian Linux, run the following command as root:
apt-get install python-twisted python-libxml2
To install optional libraries:
apt-get install python-pyopenssl python-simplejson
If you are running Arch Linux, run the following command as root:
pacman -S twisted libxml2
To install optional libraries:
pacman -S pyopenssl python-simplejson
Mac OS X¶
First, download Twisted for Mac.
Mac OS X ships an
libxml2 version too old to be used by Scrapy. Also, by
looking on the web it seems that installing
libxml2 on MacOSX is a bit of a
challenge. Here is a way to achieve this, though not acceptable on the long
Fetch the following libxml2 and libxslt packages:
Extract, build and install them both with:
./configure --with-python=/Library/Frameworks/Python.framework/Versions/2.5/ make sudo make install
/Library/Frameworks/Python.framework/Version/2.5/with your current python framework location.
Install libxml2 Python bidings with:
cd libxml2-2.7.3/python sudo make install
The libraries and modules should be installed in something like /usr/local/lib/python2.5/site-packages. Add it to your
PYTHONPATHand you are done.
libxml2library was installed properly with:
python -c 'import libxml2'
Step 3. Install Scrapy¶
There are three ways to download and install Scrapy:
Installing an official release¶
Download Scrapy from the Download page. Scrapy is distributed in two ways: a
source code tarball (for Unix and Mac OS X systems) and a Windows installer
(for Windows). If you downloaded the tarball, you can install it as any Python
tar zxf scrapy-X.X.X.tar.gz cd scrapy-X.X.X python setup.py install
If you downloaded the Windows installer, just run it.
In Windows, you may need to add the
C:\Python26\Scripts) folder to the system path by adding that directory
PATH environment variable from the Control Panel.
Installing with easy_install¶
You can install Scrapy running easy_install like this:
easy_install -U Scrapy
Installing the development version¶
If you use the development version of Scrapy, you should subscribe to the mailing lists to get notified of any changes to the API.
Check out the latest development code from the Mercurial repository (you need to install Mercurial_ first):
hg clone http://hg.scrapy.org/scrapy scrapy-trunk
Add Scrapy to your Python path
If you’re on Linux, Mac or any Unix-like system, you can make a symbolic link to your system
site-packagesdirectory like this:
ln -s /path/to/scrapy-trunk/scrapy SITE-PACKAGES/scrapy
SITE-PACKAGESis the location of your system
site-packagesdirectory. To find this out execute the following:
python -c "from distutils.sysconfig import get_python_lib; print get_python_lib()"
Alternatively, you can define your
PYTHONPATHenvironment variable so that it includes the
scrapy-trunkdirectory. This solution also works on Windows systems, which don’t support symbolic links. (Environment variables can be defined on Windows systems from the Control Panel).
Windows example (from command line, but you should probably use the Control Panel):
On Unix-like systems, create a symbolic link to the file
scrapy-trunk/bin/scrapyin a directory on your system path, such as
/usr/local/bin. For example:
ln -s `pwd`/scrapy-trunk/bin/scrapy /usr/local/bin
This simply lets you type
scrapyfrom within any directory, rather than having to qualify the command with the full path to the file.
On Windows systems, the same result can be achieved by copying the file
scrapy-trunk/bin/scrapyto somewhere on your system path, for example
C:\Python25\Scripts, which is customary for Python scripts.