# Scrapy 2.18 documentation Scrapy is a fast high-level [web crawling](https://en.wikipedia.org/wiki/Web_crawler) and [web scraping](https://en.wikipedia.org/wiki/Web_scraping) framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing. ## Getting help Having trouble? We’d like to help! * Try the [FAQ](faq.md) – it’s got answers to some common questions. * Looking for specific information? Try the [Index](genindex.md) or [Module Index](py-modindex.md). * Ask or search questions in [StackOverflow using the scrapy tag](https://stackoverflow.com/tags/scrapy). * Ask or search questions in the [Scrapy subreddit](https://www.reddit.com/r/scrapy/). * Search for questions on the archives of the [scrapy-users mailing list](https://groups.google.com/forum/#!forum/scrapy-users). * Ask a question in the [#scrapy IRC channel](irc://irc.freenode.net/scrapy). * Report bugs with Scrapy in our [issue tracker](https://github.com/scrapy/scrapy/issues). * Join the Discord community [Scrapy Discord](https://discord.com/invite/mv3yErfpvq). ## First steps [Scrapy at a glance](intro/overview.md) : Understand what Scrapy is and how it can help you. [Installation guide](intro/install.md) : Get Scrapy installed on your computer. [Scrapy Tutorial](intro/tutorial.md) : Write your first Scrapy project. [Examples](intro/examples.md) : Learn more by playing with a pre-made Scrapy project. ## Basic concepts [Scrapy’s building blocks](topics/concepts.md) : Get a quick map of Scrapy’s main concepts and when to use each. [Command line tool](topics/commands.md) : Learn about the command-line tool used to manage your Scrapy project. [Spiders](topics/spiders.md) : Write the rules to crawl your websites. [Selectors](topics/selectors.md) : Extract the data from web pages using XPath. [Items](topics/items.md) : Define the data you want to scrape. [Item Loaders](topics/loaders.md) : Populate your items with the extracted data. [Scrapy shell](topics/shell.md) : Test your extraction code in an interactive environment. [Item Pipeline](topics/item-pipeline.md) : Post-process and store your scraped data. [Feed exports](topics/feed-exports.md) : Output your scraped data using different formats and storages. [Requests and Responses](topics/request-response.md) : Understand the classes used to represent HTTP requests and responses. [Cookies](topics/cookies.md) : Send and receive cookies. [Link Extractors](topics/link-extractors.md) : Convenient classes to extract links to follow from pages. [Settings](topics/settings.md) : Learn how to configure Scrapy and see all [available settings](topics/settings.md#topics-settings-ref). [Exceptions](topics/exceptions.md) : See all available exceptions and their meaning. ## Built-in services [Logging](topics/logging.md) : Learn how to use Python’s built-in logging on Scrapy. [Stats Collection](topics/stats.md) : Collect statistics about your scraping crawler. [Telnet Console](topics/telnetconsole.md) : Inspect a running crawler using a built-in Python console. ## Solving specific problems [Frequently Asked Questions](faq.md) : Get answers to most frequently asked questions. [Debugging Spiders](topics/debug.md) : Learn how to debug common problems of your Scrapy spider. [Spiders Contracts](topics/contracts.md) : Learn how to use contracts for testing your spiders. [Common Practices](topics/practices.md) : Get familiar with some Scrapy common practices. [Security](topics/security.md) : Understand the security implications of Scrapy defaults and how to harden them. [Optimization](topics/optimize.md) : Find the bottleneck of your crawls and learn how to address it. [Using your browser’s Developer Tools for scraping](topics/developer-tools.md) : Learn how to scrape with your browser’s developer tools. [Selecting dynamically-loaded content](topics/dynamic-content.md) : Read webpage data that is loaded dynamically. [Debugging memory leaks](topics/leaks.md) : Learn how to find and get rid of memory leaks in your crawler. [Downloading and processing files and images](topics/media-pipeline.md) : Download files and/or images associated with your scraped items. [Deploying Spiders](topics/deploy.md) : Deploying your Scrapy spiders and run them in a remote server. [AutoThrottle extension](topics/autothrottle.md) : Adjust crawl rate dynamically based on load. [Benchmarking](topics/benchmarking.md) : Check how Scrapy performs on your hardware. [Jobs: pausing and resuming crawls](topics/jobs.md) : Learn how to pause and resume crawls for large spiders. [Coroutines](topics/coroutines.md) : Use the [coroutine syntax](https://docs.python.org/3/reference/compound_stmts.html#async). [asyncio](topics/asyncio.md) : Use [`asyncio`](https://docs.python.org/3/library/asyncio.html#module-asyncio) and [`asyncio`](https://docs.python.org/3/library/asyncio.html#module-asyncio)-powered libraries. ## Extending Scrapy [Architecture overview](topics/architecture.md) : Understand the Scrapy architecture. [Add-ons](topics/addons.md) : Enable and configure third-party extensions. [Downloader Middleware](topics/downloader-middleware.md) : Customize how pages get requested and downloaded. [Spider Middleware](topics/spider-middleware.md) : Customize the input and output of your spiders. [Extensions](topics/extensions.md) : Extend Scrapy with your custom functionality [Signals](topics/signals.md) : See all available signals and how to work with them. [Scheduler](topics/scheduler.md) : Understand the scheduler component. [Item Exporters](topics/exporters.md) : Quickly export your scraped items to a file (XML, CSV, etc). [Download handlers](topics/download-handlers.md) : Customize how requests are downloaded or add support for new URL schemes. [Components](topics/components.md) : Learn the common API and some good practices when building custom Scrapy components. [Core API](topics/api.md) : Use it on extensions and middlewares to extend Scrapy functionality. ## All the rest [Release notes](https://docs.scrapy.org/en/latest/news.md) : See what has changed in recent Scrapy versions. [Contributing to Scrapy](https://docs.scrapy.org/en/latest/contributing.md) : Learn how to contribute to the Scrapy project. [Versioning and API stability](versioning.md) : Understand Scrapy versioning and API stability.