Older Scrapy 2.x release notes

Scrapy 2.13.4 (2025-11-17)

Security bug fixes

  • Improved protection against decompression bombs in HttpCompressionMiddleware for responses compressed using the br and deflate methods: if a single compressed chunk would be larger than the response size limit (see DOWNLOAD_MAXSIZE) when decompressed, decompression is no longer carried out. This is especially important for the br (Brotli) method that can provide a very high compression ratio. Please, see the CVE-2025-6176 and GHSA-2qfp-q593-8484 security advisories for more information. (#7134)

Modified requirements

  • The minimum supported version of the optional brotli package is now 1.2.0. (#7134)

  • The brotlicffi and brotlipy packages can no longer be used to decompress Brotli-compressed responses. Please install the brotli package instead. (#7134)

Other changes

  • Restricted the maximum supported Twisted version to 25.5.0, as Scrapy currently uses some private APIs changed in later Twisted versions. (#7142)

  • Stopped setting the COVERAGE_CORE environment variable in tests, it didn’t have an effect but caused the coverage module to produce a warning or an error. (#7137)

  • Removed the documentation build dependency on the deprecated sphinx-hoverxref module. (#6786, #6922)

Scrapy 2.13.3 (2025-07-02)

Scrapy 2.13.2 (2025-06-09)

  • Fixed a bug introduced in Scrapy 2.13.0 that caused results of request errbacks to be ignored when the errback was called because of a downloader error. (#6861, #6863)

  • Added a note about the behavior change of scrapy.utils.reactor.is_asyncio_reactor_installed to its docs and to the “Backward-incompatible changes” section of the Scrapy 2.13.0 release notes. (#6866)

  • Improved the message in the exception raised by scrapy.utils.test.get_reactor_settings() when there is no reactor installed. (#6866)

  • Updated the scrapy.crawler.CrawlerRunner examples in Common Practices to install the reactor explicitly, to fix reactor-related errors with Scrapy 2.13.0 and later. (#6865)

  • Fixed scrapy fetch not working with scrapy-poet. (#6872)

  • Fixed an exception produced by scrapy.core.engine.ExecutionEngine when it’s closed before being fully initialized. (#6857, #6867)

  • Improved the README and updated the Scrapy logo in it. (#6831, #6833, #6839)

  • Restricted the Twisted version used in tests to below 25.5.0, as some tests fail with 25.5.0. (#6878, #6882)

  • Updated type hints for Twisted 25.5.0 changes. (#6882)

  • Removed the old artwork. (#6874)

Scrapy 2.13.1 (2025-05-28)

  • Gave callback requests precedence over start requests when priority values are the same.

    This makes the changes from 2.13.0 to start request handling more intuitive and backward compatible. For scenarios where all requests have the same priorities, in 2.13.0 all start requests were sent before the first callback request. In 2.13.1, the same as in 2.12 and lower, start requests are only sent when there are not enough pending callback requests to reach concurrency limits.

    (#6828)

  • Added a deepwiki badge to the README. (#6793)

  • Fixed a typo in the code example of Delaying start request iteration. (#6812, #6815)

  • Fixed a typo in the Supported callables section of the documentation. (#6822)

  • Made this page more prominently listed in PyPI project links. (#6826)

Scrapy 2.13.0 (2025-05-08)

Highlights:

  • The asyncio reactor is now enabled by default

  • Replaced start_requests() (sync) with start() (async) and changed how it is iterated

  • Added the allow_offsite request meta key

  • Spider middlewares that don’t support asynchronous spider output are deprecated

  • Added a base class for universal spider middlewares

Modified requirements

  • Dropped support for PyPy 3.9. (#6613)

  • Added support for PyPy 3.11. (#6697)

Backward-incompatible changes

  • The default value of the TWISTED_REACTOR setting was changed from None to "twisted.internet.asyncioreactor.AsyncioSelectorReactor". This value was used in newly generated projects since Scrapy 2.7.0 but now existing projects that don’t explicitly set this setting will also use the asyncio reactor. You can change this setting in your project to use a different reactor. (#6659, #6713)

  • The iteration of start requests and items no longer stops once there are requests in the scheduler, and instead runs continuously until all start requests have been scheduled.

    To reproduce the previous behavior, see Delaying start request iteration. (#6729)

  • An unhandled exception from the open_spider() method of a spider middleware no longer stops the crawl. (#6729)

  • In scrapy.core.engine.ExecutionEngine:

    • The second parameter of open_spider(), start_requests, has been removed. The start requests are determined by the spider parameter instead (see start()).

    • The slot attribute has been renamed to _slot and should not be used.

    (#6729)

  • In scrapy.core.engine, the Slot class has been renamed to _Slot and should not be used. (#6729)

  • The slot telnet variable has been removed. (#6729)

  • In scrapy.core.spidermw.SpiderMiddlewareManager, process_start_requests() has been replaced by process_start(). (#6729)

  • The scrape_func callable passed to scrapy.core.spidermw.SpiderMiddlewareManager.scrape_response() is now called with 2 parameters, response and request, instead of 3, and must return a Deferred instead of an iterable. (#6787)

  • The now-deprecated start_requests() method, when it returns an iterable instead of being defined as a generator, is now executed after the scheduler instance has been created. (#6729)

  • When using JOBDIR, start requests are now serialized into their own, s-suffixed priority folders. You can set SCHEDULER_START_DISK_QUEUE to None or "" to change that, but the side effects may be undesirable. See SCHEDULER_START_DISK_QUEUE for details. (#6729)

  • The URL length limit, set by the URLLENGTH_LIMIT setting, is now also enforced for start requests. (#6777)

  • Calling scrapy.utils.reactor.is_asyncio_reactor_installed without an installed reactor now raises an exception instead of installing a reactor. This shouldn’t affect normal Scrapy use cases, but it may affect 3rd-party test suites that use Scrapy internals such as Crawler and don’t install a reactor explicitly. If you are affected by this change, you most likely need to install the reactor before running Scrapy code that expects it to be installed. (#6732, #6735)

  • The from_settings() method of UrlLengthMiddleware, deprecated in Scrapy 2.12.0, is removed earlier than the usual deprecation period (this was needed because after the introduction of the BaseSpiderMiddleware base class and switching built-in spider middlewares to it those middlewares need the Crawler instance at run time). Please use from_crawler() instead. (#6693)

  • scrapy.utils.url.escape_ajax() is no longer called when a Request instance is created. It was only useful for websites supporting the _escaped_fragment_ feature which most modern websites don’t support. If you still need this you can modify the URLs before passing them to Request. (#6523, #6651)

Deprecation removals

  • Removed old deprecated name aliases for some signals:

    • stats_spider_opened (use spider_opened instead)

    • stats_spider_closing and stats_spider_closed (use spider_closed instead)

    • item_passed (use item_scraped instead)

    • request_received (use request_scheduled instead)

    (#6654, #6655)

Deprecations

  • The start_requests() method of Spider is deprecated, use start() instead, or both to maintain support for lower Scrapy versions. (#456, #3477, #4467, #5627, #6729)

  • The process_start_requests() method of spider middlewares is deprecated, use process_start() instead, or both to maintain support for lower Scrapy versions. (#456, #3477, #4467, #5627, #6729)

  • The __init__ method of priority queue classes (see SCHEDULER_PRIORITY_QUEUE) should now support a keyword-only start_queue_cls parameter. (#6752)

  • Spider middlewares that don’t support asynchronous spider output are deprecated. The async iterable downgrading feature, needed for using such middlewares with asynchronous callbacks and with other spider middlewares that produce asynchronous iterables, is also deprecated. Please update all such middlewares to support asynchronous spider output. (#6664)

  • Functions that were imported from w3lib.url and re-exported in scrapy.utils.url are now deprecated, you should import them from w3lib.url directly. They are:

    • scrapy.utils.url.add_or_replace_parameter()

    • scrapy.utils.url.add_or_replace_parameters()

    • scrapy.utils.url.any_to_uri()

    • scrapy.utils.url.canonicalize_url()

    • scrapy.utils.url.file_uri_to_path()

    • scrapy.utils.url.is_url()

    • scrapy.utils.url.parse_data_uri()

    • scrapy.utils.url.parse_url()

    • scrapy.utils.url.path_to_file_uri()

    • scrapy.utils.url.safe_download_url()

    • scrapy.utils.url.safe_url_string()

    • scrapy.utils.url.url_query_cleaner()

    • scrapy.utils.url.url_query_parameter()

    (#4577, #6583, #6586)

  • HTTP/1.0 support code is deprecated. It was disabled by default and couldn’t be used together with HTTP/1.1. If you still need it, you should write your own download handler or copy the code from Scrapy. The deprecations include:

    • scrapy.core.downloader.handlers.http10.HTTP10DownloadHandler

    • scrapy.core.downloader.webclient.ScrapyHTTPClientFactory

    • scrapy.core.downloader.webclient.ScrapyHTTPPageGetter

    • Overriding scrapy.core.downloader.contextfactory.ScrapyClientContextFactory.getContext()

    (#6634)

  • The following modules and functions used only in tests are deprecated:

    • the scrapy.utils.testproc module

    • the scrapy.utils.testsite module

    • scrapy.utils.test.assert_gcs_environ()

    • scrapy.utils.test.get_ftp_content_and_delete()

    • scrapy.utils.test.get_gcs_content_and_delete()

    • scrapy.utils.test.mock_google_cloud_storage()

    • scrapy.utils.test.skip_if_no_boto()

    If you need to use them in your tests or code, you can copy the code from Scrapy. (#6696)

  • scrapy.utils.test.TestSpider is deprecated. If you need an empty spider class you can use scrapy.utils.spider.DefaultSpider or create your own subclass of scrapy.Spider. (#6678)

  • scrapy.downloadermiddlewares.ajaxcrawl.AjaxCrawlMiddleware is deprecated. It was disabled by default and isn’t useful for most of the existing websites. (#6523, #6651, #6656)

  • scrapy.utils.url.escape_ajax() is deprecated. (#6523, #6651)

  • scrapy.spiders.init.InitSpider is deprecated. If you find it useful, you can copy its code from Scrapy. (#6708, #6714)

  • scrapy.utils.versions.scrapy_components_versions() is deprecated, use scrapy.utils.versions.get_versions() instead. (#6582)

  • BaseDupeFilter.log() is deprecated. It does nothing and shouldn’t be called. (#4151)

  • Passing the spider argument to the following methods of Scraper is deprecated:

    • close_spider()

    • enqueue_scrape()

    • handle_spider_error()

    • handle_spider_output()

    (#6764)

New features

Improvements

  • Removed or postponed some calls of itemadapter.is_item() to increase performance. (#6719)

  • Improved the error message when running a scrapy command that requires a project (such as scrapy crawl) outside of a project directory. (#2349, #3426)

  • Added an empty ADDONS setting to the settings.py template for new projects. (#6587)

Bug fixes

  • Yielding an item from Spider.start or from SpiderMiddleware.process_start no longer delays the next iteration of starting requests and items by up to 5 seconds. (#6729)

  • Fixed calculation of items_per_minute and responses_per_minute stats. (#6599)

  • Fixed an error initializing scrapy.extensions.feedexport.GCSFeedStorage. (#6617, #6628)

  • Fixed an error running scrapy bench. (#6632, #6633)

  • Fixed duplicated log messages about the reactor and the event loop. (#6636, #6657)

  • Fixed resolving type annotations of SitemapSpider._parse_sitemap() at run time, required by tools such as scrapy-poet. (#6665, #6671)

  • Calling scrapy.utils.reactor.is_asyncio_reactor_installed without an installed reactor now raises an exception instead of installing a reactor. (#6732, #6735)

  • Restored support for the x-gzip content encoding. (#6618)

Documentation

Packaging

Quality assurance

Scrapy 2.12.0 (2024-11-18)

Highlights:

Modified requirements

  • Dropped support for Python 3.8. (#6466, #6472)

  • Added support for Python 3.13. (#6166)

  • Minimum versions increased for these dependencies:

  • Removed setuptools from the dependency list. (#6487)

Backward-incompatible changes

  • User-defined cookies for HTTPS requests will have the secure flag set to True unless it’s set to False explicitly. This is important when these cookies are reused in HTTP requests, e.g. after a redirect to an HTTP URL. (#6357)

  • The Reppy-based robots.txt parser, scrapy.robotstxt.ReppyRobotParser, was removed, as it doesn’t support Python 3.9+. (#5230, #6099, #6499)

  • The initialization API of scrapy.pipelines.media.MediaPipeline and its subclasses was improved and it’s possible that some previously working usage scenarios will no longer work. It can only affect you if you define custom subclasses of MediaPipeline or create instances of these pipelines via from_settings() or __init__() calls instead of from_crawler() calls.

    Previously, MediaPipeline.from_crawler() called the from_settings() method if it existed or the __init__() method otherwise, and then did some additional initialization using the crawler instance. If the from_settings() method existed (like in FilesPipeline) it called __init__() to create the instance. It wasn’t possible to override from_crawler() without calling MediaPipeline.from_crawler() from it which, in turn, couldn’t be called in some cases (including subclasses of FilesPipeline).

    Now, in line with the general usage of from_crawler() and from_settings() and the deprecation of the latter the recommended initialization order is the following one:

    • All __init__() methods should take a crawler argument. If they also take a settings argument they should ignore it, using crawler.settings instead. When they call __init__() of the base class they should pass the crawler argument to it too.

    • A from_settings() method shouldn’t be defined. Class-specific initialization code should go into either an overridden from_crawler() method or into __init__().

    • It’s now possible to override from_crawler() and it’s not necessary to call MediaPipeline.from_crawler() in it if other recommendations were followed.

    • If pipeline instances were created with from_settings() or __init__() calls (which wasn’t supported even before, as it missed important initialization code), they should now be created with from_crawler() calls.

    (#6540)

  • The response_body argument of ImagesPipeline.convert_image is now positional-only, as it was changed from optional to required. (#6500)

  • The convert argument of scrapy.utils.conf.build_component_list() is now positional-only, as the preceding argument (custom) was removed. (#6500)

  • The overwrite_output argument of scrapy.utils.conf.feed_process_params_from_cli() is now positional-only, as the preceding argument (output_format) was removed. (#6500)

Deprecation removals

  • Removed the scrapy.utils.request.request_fingerprint() function, deprecated in Scrapy 2.7.0. (#6212, #6213)

  • Removed support for value "2.6" of setting REQUEST_FINGERPRINTER_IMPLEMENTATION, deprecated in Scrapy 2.7.0. (#6212, #6213)

  • RFPDupeFilter subclasses now require supporting the fingerprinter parameter in their __init__ method, introduced in Scrapy 2.7.0. (#6102, #6113)

  • Removed the scrapy.downloadermiddlewares.decompression module, deprecated in Scrapy 2.7.0. (#6100, #6113)

  • Removed the scrapy.utils.response.response_httprepr() function, deprecated in Scrapy 2.6.0. (#6111, #6116)

  • Spiders with spider-level HTTP authentication, i.e. with the http_user or http_pass attributes, must now define http_auth_domain as well, which was introduced in Scrapy 2.5.1. (#6103, #6113)

  • Media pipelines methods file_path(), file_downloaded(), get_images(), image_downloaded(), media_downloaded(), media_to_download(), and thumb_path() must now support an item parameter, added in Scrapy 2.4.0. (#6107, #6113)

  • The __init__() and from_crawler() methods of feed storage backend classes must now support the keyword-only feed_options parameter, introduced in Scrapy 2.4.0. (#6105, #6113)

  • Removed the scrapy.loader.common and scrapy.loader.processors modules, deprecated in Scrapy 2.3.0. (#6106, #6113)

  • Removed the scrapy.utils.misc.extract_regex() function, deprecated in Scrapy 2.3.0. (#6106, #6113)

  • Removed the scrapy.http.JSONRequest class, replaced with JsonRequest in Scrapy 1.8.0. (#6110, #6113)

  • scrapy.utils.log.logformatter_adapter no longer supports missing args, level, or msg parameters, and no longer supports a format parameter, all scenarios that were deprecated in Scrapy 1.0.0. (#6109, #6116)

  • A custom class assigned to the SPIDER_LOADER_CLASS setting that does not implement the ISpiderLoader interface will now raise a zope.interface.verify.DoesNotImplement exception at run time. Non-compliant classes have been triggering a deprecation warning since Scrapy 1.0.0. (#6101, #6113)

  • Removed the --output-format/-t command line option, deprecated in Scrapy 2.1.0. -O <URI>:<FORMAT> should be used instead. (#6500)

  • Running crawl() more than once on the same Crawler instance, deprecated in Scrapy 2.11.0, now raises an exception. (#6500)

  • Subclassing HttpCompressionMiddleware without support for the crawler argument in __init__() and without a custom from_crawler() method, deprecated in Scrapy 2.5.0, is no longer allowed. (#6500)

  • Removed the EXCEPTIONS_TO_RETRY attribute of RetryMiddleware, deprecated in Scrapy 2.10.0. (#6500)

  • Removed support for S3 feed exports without the boto3 package installed, deprecated in Scrapy 2.10.0. (#6500)

  • Removed the scrapy.extensions.feedexport._FeedSlot class, deprecated in Scrapy 2.10.0. (#6500)

  • Removed the scrapy.pipelines.images.NoimagesDrop exception, deprecated in Scrapy 2.8.0. (#6500)

  • The response_body argument of ImagesPipeline.convert_image is now required, not passing it was deprecated in Scrapy 2.8.0. (#6500)

  • Removed the custom argument of scrapy.utils.conf.build_component_list(), deprecated in Scrapy 2.10.0. (#6500)

  • Removed the scrapy.utils.reactor.get_asyncio_event_loop_policy() function, deprecated in Scrapy 2.9.0. Use asyncio.get_event_loop() and related standard library functions instead. (#6500)

Deprecations

  • The from_settings() methods of the Scrapy components that have them are now deprecated. from_crawler() should now be used instead. Affected components:

    (#6540)

  • It’s now deprecated to have a from_settings() method but no from_crawler() method in 3rd-party Scrapy components. You can define a simple from_crawler() method that calls cls.from_settings(crawler.settings) to fix this if you don’t want to refactor the code. Note that if you have a from_crawler() method Scrapy will not call the from_settings() method so the latter can be removed. (#6540)

  • The initialization API of scrapy.pipelines.media.MediaPipeline and its subclasses was improved and some old usage scenarios are now deprecated (see also the “Backward-incompatible changes” section). Specifically:

    • It’s deprecated to define an __init__() method that doesn’t take a crawler argument.

    • It’s deprecated to call an __init__() method without passing a crawler argument. If it’s passed, it’s also deprecated to pass a settings argument, which will be ignored anyway.

    • Calling from_settings() is deprecated, use from_crawler() instead.

    • Overriding from_settings() is deprecated, override from_crawler() instead.

    (#6540)

  • The REQUEST_FINGERPRINTER_IMPLEMENTATION setting is now deprecated. (#6212, #6213)

  • The scrapy.utils.misc.create_instance() function is now deprecated, use scrapy.utils.misc.build_from_crawler() instead. (#5523, #5884, #6162, #6169, #6540)

  • scrapy.core.downloader.Downloader._get_slot_key() is deprecated, use scrapy.core.downloader.Downloader.get_slot_key() instead. (#6340, #6352)

  • scrapy.utils.defer.process_chain_both() is now deprecated. (#6397)

  • scrapy.twisted_version is now deprecated, you should instead use twisted.version directly (but note that it’s an incremental.Version object, not a tuple). (#6509, #6512)

  • scrapy.utils.python.flatten() and scrapy.utils.python.iflatten() are now deprecated. (#6517, #6519)

  • scrapy.utils.python.equal_attributes() is now deprecated. (#6517, #6519)

  • scrapy.utils.request.request_authenticate() is now deprecated, you should instead just set the Authorization header directly. (#6517, #6519)

  • scrapy.utils.serialize.ScrapyJSONDecoder is now deprecated, it didn’t contain any code since Scrapy 1.0.0. (#6517, #6519)

  • scrapy.utils.test.assert_samelines() is now deprecated. (#6517, #6519)

  • scrapy.extensions.feedexport.build_storage() is now deprecated. You can instead call the builder callable directly. (#6540)

  • scrapy.utils.misc.md5sum() is now deprecated. (#6264)

New features

Improvements

Bug fixes

  • MediaPipeline is now an abstract class and its methods that were expected to be overridden in subclasses are now abstract methods. (#6365, #6368)

  • Fixed handling of invalid @-prefixed lines in contract extraction. (#6383, #6388)

  • Importing scrapy.extensions.telnet no longer installs the default reactor. (#6432)

  • Reduced log verbosity for dropped requests that was increased in 2.11.2. (#6433, #6475)

Documentation

Quality assurance

Other

  • Issue tracker improvements. (#6066)

Scrapy 2.11.2 (2024-05-14)

Security bug fixes

  • Redirects to non-HTTP protocols are no longer followed. Please, see the 23j4-mw76-5v7h security advisory for more information. (#457)

  • The Authorization header is now dropped on redirects to a different scheme (http:// or https://) or port, even if the domain is the same. Please, see the 4qqq-9vqf-3h3f security advisory for more information.

  • When using system proxy settings that are different for http:// and https://, redirects to a different URL scheme will now also trigger the corresponding change in proxy settings for the redirected request. Please, see the jm3v-qxmh-hxwv security advisory for more information. (#767)

  • Spider.allowed_domains is now enforced for all requests, and not only requests from spider callbacks. (#1042, #2241, #6358)

  • xmliter_lxml() no longer resolves XML entities. (#6265)

  • defusedxml is now used to make scrapy.http.request.rpc.XmlRpcRequest more secure. (#6250, #6251)

Deprecations

Bug fixes

Documentation

Quality assurance

Scrapy 2.11.1 (2024-02-14)

Highlights:

  • Security bug fixes.

  • Support for Twisted >= 23.8.0.

  • Documentation improvements.

Security bug fixes

Modified requirements

  • The Twisted dependency is no longer restricted to < 23.8.0. (#6024, #6064, #6142)

Bug fixes

  • The OS signal handling code was refactored to no longer use private Twisted functions. (#6024, #6064, #6112)

Documentation

Quality assurance

  • Added Python 3.12 to the CI configuration, re-enabled tests that were disabled when the pre-release support was added. (#5985, #6083, #6098)

  • Fixed a test issue on PyPy 7.3.14. (#6204, #6205)

Scrapy 2.11.0 (2023-09-18)

Highlights:

Backward-incompatible changes

  • Most of the initialization of scrapy.crawler.Crawler instances is now done in crawl(), so the state of instances before that method is called is now different compared to older Scrapy versions. We do not recommend using the Crawler instances before crawl() is called. (#6038)

  • scrapy.Spider.from_crawler() is now called before the initialization of various components previously initialized in scrapy.crawler.Crawler.__init__() and before the settings are finalized and frozen. This change was needed to allow changing the settings in scrapy.Spider.from_crawler(). If you want to access the final setting values and the initialized Crawler attributes in the spider code as early as possible you can do this in scrapy.Spider.start_requests() or in a handler of the engine_started signal. (#6038)

  • The TextResponse.json method now requires the response to be in a valid JSON encoding (UTF-8, UTF-16, or UTF-32). If you need to deal with JSON documents in an invalid encoding, use json.loads(response.text) instead. (#6016)

  • PythonItemExporter used the binary output by default but it no longer does. (#6006, #6007)

Deprecation removals

  • Removed the binary export mode of PythonItemExporter, deprecated in Scrapy 1.1.0. (#6006, #6007)

    Note

    If you are using this Scrapy version on Scrapy Cloud with a stack that includes an older Scrapy version and get a “TypeError: Unexpected options: binary” error, you may need to add scrapinghub-entrypoint-scrapy >= 0.14.1 to your project requirements or switch to a stack that includes Scrapy 2.11.

  • Removed the CrawlerRunner.spiders attribute, deprecated in Scrapy 1.0.0, use CrawlerRunner.spider_loader instead. (#6010)

  • The scrapy.utils.response.response_httprepr() function, deprecated in Scrapy 2.6.0, has now been removed. (#6111)

Deprecations

New features

Bug fixes

Documentation

  • Updated a deprecated function call in a pipeline example. (#6008, #6009)

Quality assurance

Scrapy 2.10.1 (2023-08-30)

Marked Twisted >= 23.8.0 as unsupported. (#6024, #6026)

Scrapy 2.10.0 (2023-08-04)

Highlights:

  • Added Python 3.12 support, dropped Python 3.7 support.

  • The new add-ons framework simplifies configuring 3rd-party components that support it.

  • Exceptions to retry can now be configured.

  • Many fixes and improvements for feed exports.

Modified requirements

  • Dropped support for Python 3.7. (#5953)

  • Added support for the upcoming Python 3.12. (#5984)

  • Minimum versions increased for these dependencies:

  • pkg_resources is no longer used. (#5956, #5958)

  • boto3 is now recommended instead of botocore for exporting to S3. (#5833).

Backward-incompatible changes

  • The value of the FEED_STORE_EMPTY setting is now True instead of False. In earlier Scrapy versions empty files were created even when this setting was False (which was a bug that is now fixed), so the new default should keep the old behavior. (#872, #5847)

Deprecation removals

  • When a function is assigned to the FEED_URI_PARAMS setting, returning None or modifying the params input parameter, deprecated in Scrapy 2.6, is no longer supported. (#5994, #5996)

  • The scrapy.utils.reqser module, deprecated in Scrapy 2.6, is removed. (#5994, #5996)

  • The scrapy.squeues classes PickleFifoDiskQueueNonRequest, PickleLifoDiskQueueNonRequest, MarshalFifoDiskQueueNonRequest, and MarshalLifoDiskQueueNonRequest, deprecated in Scrapy 2.6, are removed. (#5994, #5996)

  • The property open_spiders and the methods has_capacity and schedule of scrapy.core.engine.ExecutionEngine, deprecated in Scrapy 2.6, are removed. (#5994, #5998)

  • Passing a spider argument to the spider_is_idle(), crawl() and download() methods of scrapy.core.engine.ExecutionEngine, deprecated in Scrapy 2.6, is no longer supported. (#5994, #5998)

Deprecations

  • scrapy.utils.datatypes.CaselessDict is deprecated, use scrapy.utils.datatypes.CaseInsensitiveDict instead. (#5146)

  • Passing the custom argument to scrapy.utils.conf.build_component_list() is deprecated, it was used in the past to merge FOO and FOO_BASE setting values but now Scrapy uses scrapy.settings.BaseSettings.getwithbase() to do the same. Code that uses this argument and cannot be switched to getwithbase() can be switched to merging the values explicitly. (#5726, #5923)

New features

Bug fixes

  • Fixed creating empty feeds even with FEED_STORE_EMPTY=False. (#872, #5847)

  • Fixed using absolute Windows paths when specifying output files. (#5969, #5971)

  • Fixed problems with uploading large files to S3 by switching to multipart uploads (requires boto3). (#960, #5735, #5833)

  • Fixed the JSON exporter writing extra commas when some exceptions occur. (#3090, #5952)

  • Fixed the “read of closed file” error in the CSV exporter. (#5043, #5705)

  • Fixed an error when a component added by the class object throws NotConfigured with a message. (#5950, #5992)

  • Added the missing scrapy.settings.BaseSettings.pop() method. (#5959, #5960, #5963)

  • Added CaseInsensitiveDict as a replacement for CaselessDict that fixes some API inconsistencies. (#5146)

Documentation

Quality assurance

  • Added support for running tests against the installed Scrapy version. (#4914, #5949)

  • Extended typing hints. (#5925, #5977)

  • Fixed the test_utils_asyncio.AsyncioTest.test_set_asyncio_event_loop test. (#5951)

  • Fixed the test_feedexport.BatchDeliveriesTest.test_batch_path_differ test on Windows. (#5847)

  • Enabled CI runs for Python 3.11 on Windows. (#5999)

  • Simplified skipping tests that depend on uvloop. (#5984)

  • Fixed the extra-deps-pinned tox env. (#5948)

  • Implemented cleanups. (#5965, #5986)

Scrapy 2.9.0 (2023-05-08)

Highlights:

  • Per-domain download settings.

  • Compatibility with new cryptography and new parsel.

  • JMESPath selectors from the new parsel.

  • Bug fixes.

Deprecations

  • scrapy.extensions.feedexport._FeedSlot is renamed to scrapy.extensions.feedexport.FeedSlot and the old name is deprecated. (#5876)

New features

Bug fixes

Documentation

Quality assurance

  • Extended typing hints. (#5805, #5889, #5896)

  • Tests for most of the examples in the docs are now run as a part of CI, found problems were fixed. (#5816, #5826, #5919)

  • Removed usage of deprecated Python classes. (#5849)

  • Silenced include-ignored warnings from coverage. (#5820)

  • Fixed a random failure of the test_feedexport.test_batch_path_differ test. (#5855, #5898)

  • Updated docstrings to match output produced by parsel 1.8.1 so that they don’t cause test failures. (#5902, #5919)

  • Other CI and pre-commit improvements. (#5802, #5823, #5908)

Scrapy 2.8.0 (2023-02-02)

This is a maintenance release, with minor features, bug fixes, and cleanups.

Deprecation removals

  • The scrapy.utils.gz.read1 function, deprecated in Scrapy 2.0, has now been removed. Use the read1() method of GzipFile instead. (#5719)

  • The scrapy.utils.python.to_native_str function, deprecated in Scrapy 2.0, has now been removed. Use scrapy.utils.python.to_unicode() instead. (#5719)

  • The scrapy.utils.python.MutableChain.next method, deprecated in Scrapy 2.0, has now been removed. Use __next__() instead. (#5719)

  • The scrapy.linkextractors.FilteringLinkExtractor class, deprecated in Scrapy 2.0, has now been removed. Use LinkExtractor instead. (#5720)

  • Support for using environment variables prefixed with SCRAPY_ to override settings, deprecated in Scrapy 2.0, has now been removed. (#5724)

  • Support for the noconnect query string argument in proxy URLs, deprecated in Scrapy 2.0, has now been removed. We expect proxies that used to need it to work fine without it. (#5731)

  • The scrapy.utils.python.retry_on_eintr function, deprecated in Scrapy 2.3, has now been removed. (#5719)

  • The scrapy.utils.python.WeakKeyCache class, deprecated in Scrapy 2.4, has now been removed. (#5719)

  • The scrapy.utils.boto.is_botocore() function, deprecated in Scrapy 2.4, has now been removed. (#5719)

Deprecations

  • scrapy.pipelines.images.NoimagesDrop is now deprecated. (#5368, #5489)

  • ImagesPipeline.convert_image must now accept a response_body parameter. (#3055, #3689, #4753)

New features

Bug fixes

  • Enabled unsafe legacy SSL renegotiation to fix access to some outdated websites. (#5491, #5790)

  • Fixed STARTTLS-based email delivery not working with Twisted 21.2.0 and better. (#5386, #5406)

  • Fixed the finish_exporting() method of item exporters not being called for empty files. (#5537, #5758)

  • Fixed HTTP/2 responses getting only the last value for a header when multiple headers with the same name are received. (#5777)

  • Fixed an exception raised by the shell command on some cases when using asyncio. (#5740, #5742, #5748, #5759, #5760, #5771)

  • When using CrawlSpider, callback keyword arguments (cb_kwargs) added to a request in the process_request callback of a Rule will no longer be ignored. (#5699)

  • The images pipeline no longer re-encodes JPEG files. (#3055, #3689, #4753)

  • Fixed the handling of transparent WebP images by the images pipeline. (#3072, #5766, #5767)

  • scrapy.shell.inspect_response() no longer inhibits SIGINT (Ctrl+C). (#2918)

  • LinkExtractor with unique=False no longer filters out links that have identical URL and text. (#3798, #3799, #4695, #5458)

  • RobotsTxtMiddleware now ignores URL protocols that do not support robots.txt (data://, file://). (#5807)

  • Silenced the filelock debug log messages introduced in Scrapy 2.6. (#5753, #5754)

  • Fixed the output of scrapy -h showing an unintended **commands** line. (#5709, #5711, #5712)

  • Made the active project indication in the output of commands more clear. (#5715)

Documentation

Quality assurance

Scrapy 2.7.1 (2022-11-02)

New features

  • Relaxed the restriction introduced in 2.6.2 so that the Proxy-Authorization header can again be set explicitly, as long as the proxy URL in the proxy metadata has no other credentials, and for as long as that proxy URL remains the same; this restores compatibility with scrapy-zyte-smartproxy 2.1.0 and older (#5626).

Bug fixes

  • Using -O/--overwrite-output and -t/--output-format options together now produces an error instead of ignoring the former option (#5516, #5605).

  • Replaced deprecated asyncio APIs that implicitly use the current event loop with code that explicitly requests a loop from the event loop policy (#5685, #5689).

  • Fixed uses of deprecated Scrapy APIs in Scrapy itself (#5588, #5589).

  • Fixed uses of a deprecated Pillow API (#5684, #5692).

  • Improved code that checks if generators return values, so that it no longer fails on decorated methods and partial methods (#5323, #5592, #5599, #5691).

Documentation

  • Upgraded the Code of Conduct to Contributor Covenant v2.1 (#5698).

  • Fixed typos (#5681, #5694).

Quality assurance

  • Re-enabled some erroneously disabled flake8 checks (#5688).

  • Ignored harmless deprecation warnings from typing in tests (#5686, #5697).

  • Modernized our CI configuration (#5695, #5696).

Scrapy 2.7.0 (2022-10-17)

Highlights:

Modified requirements

Python 3.7 or greater is now required; support for Python 3.6 has been dropped. Support for the upcoming Python 3.11 has been added.

The minimum required version of some dependencies has changed as well:

(#5512, #5514, #5524, #5563, #5664, #5670, #5678)

Deprecations

New features

Bug fixes

Documentation

Quality assurance

Scrapy 2.6.3 (2022-09-27)

Scrapy 2.6.2 (2022-07-25)

Security bug fix:

  • When HttpProxyMiddleware processes a request with proxy metadata, and that proxy metadata includes proxy credentials, HttpProxyMiddleware sets the Proxy-Authorization header, but only if that header is not already set.

    There are third-party proxy-rotation downloader middlewares that set different proxy metadata every time they process a request.

    Because of request retries and redirects, the same request can be processed by downloader middlewares more than once, including both HttpProxyMiddleware and any third-party proxy-rotation downloader middleware.

    These third-party proxy-rotation downloader middlewares could change the proxy metadata of a request to a new value, but fail to remove the Proxy-Authorization header from the previous value of the proxy metadata, causing the credentials of one proxy to be sent to a different proxy.

    To prevent the unintended leaking of proxy credentials, the behavior of HttpProxyMiddleware is now as follows when processing a request:

    • If the request being processed defines proxy metadata that includes credentials, the Proxy-Authorization header is always updated to feature those credentials.

    • If the request being processed defines proxy metadata without credentials, the Proxy-Authorization header is removed unless it was originally defined for the same proxy URL.

      To remove proxy credentials while keeping the same proxy URL, remove the Proxy-Authorization header.

    • If the request has no proxy metadata, or that metadata is a falsy value (e.g. None), the Proxy-Authorization header is removed.

      It is no longer possible to set a proxy URL through the proxy metadata but set the credentials through the Proxy-Authorization header. Set proxy credentials through the proxy metadata instead.

Also fixes the following regressions introduced in 2.6.0:

  • CrawlerProcess supports again crawling multiple spiders (#5435, #5436)

  • Installing a Twisted reactor before Scrapy does (e.g. importing twisted.internet.reactor somewhere at the module level) no longer prevents Scrapy from starting, as long as a different reactor is not specified in TWISTED_REACTOR (#5525, #5528)

  • Fixed an exception that was being logged after the spider finished under certain conditions (#5437, #5440)

  • The --output/-o command-line parameter supports again a value starting with a hyphen (#5444, #5445)

  • The scrapy parse -h command no longer throws an error (#5481, #5482)

Scrapy 2.6.1 (2022-03-01)

Fixes a regression introduced in 2.6.0 that would unset the request method when following redirects.

Scrapy 2.6.0 (2022-03-01)

Highlights:

Security bug fixes

  • When a Request object with cookies defined gets a redirect response causing a new Request object to be scheduled, the cookies defined in the original Request object are no longer copied into the new Request object.

    If you manually set the Cookie header on a Request object and the domain name of the redirect URL is not an exact match for the domain of the URL of the original Request object, your Cookie header is now dropped from the new Request object.

    The old behavior could be exploited by an attacker to gain access to your cookies. Please, see the cjvr-mfj7-j4j8 security advisory for more information.

    Note

    It is still possible to enable the sharing of cookies between different domains with a shared domain suffix (e.g. example.com and any subdomain) by defining the shared domain suffix (e.g. example.com) as the cookie domain when defining your cookies. See the documentation of the Request class for more information.

  • When the domain of a cookie, either received in the Set-Cookie header of a response or defined in a Request object, is set to a public suffix, the cookie is now ignored unless the cookie domain is the same as the request domain.

    The old behavior could be exploited by an attacker to inject cookies from a controlled domain into your cookiejar that could be sent to other domains not controlled by the attacker. Please, see the mfjm-vh54-3f96 security advisory for more information.

Modified requirements

Backward-incompatible changes

  • The formdata parameter of FormRequest, if specified for a non-POST request, now overrides the URL query string, instead of being appended to it. (#2919, #3579)

  • When a function is assigned to the FEED_URI_PARAMS setting, now the return value of that function, and not the params input parameter, will determine the feed URI parameters, unless that return value is None. (#4962, #4966)

  • In scrapy.core.engine.ExecutionEngine, methods crawl(), download(), schedule(), and spider_is_idle() now raise RuntimeError if called before open_spider(). (#5090)

    These methods used to assume that ExecutionEngine.slot had been defined by a prior call to open_spider(), so they were raising AttributeError instead.

  • If the API of the configured scheduler does not meet expectations, TypeError is now raised at startup time. Before, other exceptions would be raised at run time. (#3559)

  • The _encoding field of serialized Request objects is now named encoding, in line with all other fields (#5130)

Deprecation removals

  • scrapy.http.TextResponse.body_as_unicode, deprecated in Scrapy 2.2, has now been removed. (#5393)

  • scrapy.item.BaseItem, deprecated in Scrapy 2.2, has now been removed. (#5398)

  • scrapy.item.DictItem, deprecated in Scrapy 1.8, has now been removed. (#5398)

  • scrapy.Spider.make_requests_from_url, deprecated in Scrapy 1.4, has now been removed. (#4178, #4356)

Deprecations

  • When a function is assigned to the FEED_URI_PARAMS setting, returning None or modifying the params input parameter is now deprecated. Return a new dictionary instead. (#4962, #4966)

  • scrapy.utils.reqser is deprecated. (#5130)

  • In scrapy.squeues, the following queue classes are deprecated: PickleFifoDiskQueueNonRequest, PickleLifoDiskQueueNonRequest, MarshalFifoDiskQueueNonRequest, and MarshalLifoDiskQueueNonRequest. You should instead use: PickleFifoDiskQueue, PickleLifoDiskQueue, MarshalFifoDiskQueue, and MarshalLifoDiskQueue. (#5117)

  • Many aspects of scrapy.core.engine.ExecutionEngine that come from a time when this class could handle multiple Spider objects at a time have been deprecated. (#5090)

    • The has_capacity() method is deprecated.

    • The schedule() method is deprecated, use crawl() or download() instead.

    • The open_spiders attribute is deprecated, use spider instead.

    • The spider parameter is deprecated for the following methods:

      • spider_is_idle()

      • crawl()

      • download()

      Instead, call open_spider() first to set the Spider object.

  • scrapy.utils.response.response_httprepr() is now deprecated. (#4972)

New features

Bug fixes

Documentation

Quality Assurance

Scrapy 2.5.1 (2021-10-05)

  • Security bug fix:

    If you use HttpAuthMiddleware (i.e. the http_user and http_pass spider attributes) for HTTP authentication, any request exposes your credentials to the request target.

    To prevent unintended exposure of authentication credentials to unintended domains, you must now additionally set a new, additional spider attribute, http_auth_domain, and point it to the specific domain to which the authentication credentials must be sent.

    If the http_auth_domain spider attribute is not set, the domain of the first request will be considered the HTTP authentication target, and authentication credentials will only be sent in requests targeting that domain.

    If you need to send the same HTTP authentication credentials to multiple domains, you can use w3lib.http.basic_auth_header() instead to set the value of the Authorization header of your requests.

    If you really want your spider to send the same HTTP authentication credentials to any domain, set the http_auth_domain spider attribute to None.

    Finally, if you are a user of scrapy-splash, know that this version of Scrapy breaks compatibility with scrapy-splash 0.7.2 and earlier. You will need to upgrade scrapy-splash to a greater version for it to continue to work.

Scrapy 2.5.0 (2021-04-06)

Highlights:

Deprecation removals

Deprecations

  • The scrapy.utils.py36 module is now deprecated in favor of scrapy.utils.asyncgen. (#4900)

New features

Bug fixes

  • Fixed installation on PyPy installing PyDispatcher in addition to PyPyDispatcher, which could prevent Scrapy from working depending on which package got imported. (#4710, #4814)

  • When inspecting a callback to check if it is a generator that also returns a value, an exception is no longer raised if the callback has a docstring with lower indentation than the following code. (#4477, #4935)

  • The Content-Length header is no longer omitted from responses when using the default, HTTP/1.1 download handler (see DOWNLOAD_HANDLERS). (#5009, #5034, #5045, #5057, #5062)

  • Setting the handle_httpstatus_all request meta key to False now has the same effect as not setting it at all, instead of having the same effect as setting it to True. (#3851, #4694)

Documentation

Quality Assurance

Scrapy 2.4.1 (2020-11-17)

  • Fixed feed exports overwrite support (#4845, #4857, #4859)

  • Fixed the AsyncIO event loop handling, which could make code hang (#4855, #4872)

  • Fixed the IPv6-capable DNS resolver CachingHostnameResolver for download handlers that call reactor.resolve (#4802, #4803)

  • Fixed the output of the genspider command showing placeholders instead of the import path of the generated spider module (#4874)

  • Migrated Windows CI from Azure Pipelines to GitHub Actions (#4869, #4876)

Scrapy 2.4.0 (2020-10-11)

Highlights:

  • Python 3.5 support has been dropped.

  • The file_path method of media pipelines can now access the source item.

    This allows you to set a download file path based on item data.

  • The new item_export_kwargs key of the FEEDS setting allows to define keyword parameters to pass to item exporter classes

  • You can now choose whether feed exports overwrite or append to the output file.

    For example, when using the crawl or runspider commands, you can use the -O option instead of -o to overwrite the output file.

  • Zstd-compressed responses are now supported if zstandard is installed.

  • In settings, where the import path of a class is required, it is now possible to pass a class object instead.

Modified requirements

Backward-incompatible changes

  • CookiesMiddleware once again discards cookies defined in Request.headers.

    We decided to revert this bug fix, introduced in Scrapy 2.2.0, because it was reported that the current implementation could break existing code.

    If you need to set cookies for a request, use the Request.cookies parameter.

    A future version of Scrapy will include a new, better implementation of the reverted bug fix.

    (#4717, #4823)

Deprecation removals

  • scrapy.extensions.feedexport.S3FeedStorage no longer reads the values of access_key and secret_key from the running project settings when they are not passed to its __init__ method; you must either pass those parameters to its __init__ method or use S3FeedStorage.from_crawler (#4356, #4411, #4688)

  • Rule.process_request no longer admits callables which expect a single request parameter, rather than both request and response (#4818)

Deprecations

New features

Bug fixes

  • The genspider command no longer overwrites existing files unless the --force option is used (#4561, #4616, #4623)

  • Cookies with an empty value are no longer considered invalid cookies (#4772)

  • The runspider command now supports files with the .pyw file extension (#4643, #4646)

  • The HttpProxyMiddleware middleware now simply ignores unsupported proxy values (#3331, #4778)

  • Checks for generator callbacks with a return statement no longer warn about return statements in nested functions (#4720, #4721)

  • The system file mode creation mask no longer affects the permissions of files generated using the startproject command (#4722)

  • scrapy.utils.iterators.xmliter now supports namespaced node names (#861, #4746)

  • Request objects can now have about: URLs, which can work when using a headless browser (#4835)

Documentation

Quality assurance

Scrapy 2.3.0 (2020-08-04)

Highlights:

Deprecation removals

  • Removed the following classes and their parent modules from scrapy.linkextractors:

    • htmlparser.HtmlParserLinkExtractor

    • regex.RegexLinkExtractor

    • sgml.BaseSgmlLinkExtractor

    • sgml.SgmlLinkExtractor

    Use LinkExtractor instead (#4356, #4679)

Deprecations

  • The scrapy.utils.python.retry_on_eintr function is now deprecated (#4683)

New features

Bug fixes

Documentation

Quality assurance

  • The base implementation of item loaders has been moved into itemloaders (#4005, #4516)

  • Fixed a silenced error in some scheduler tests (#4644, #4645)

  • Renewed the localhost certificate used for SSL tests (#4650)

  • Removed cookie-handling code specific to Python 2 (#4682)

  • Stopped using Python 2 unicode literal syntax (#4704)

  • Stopped using a backlash for line continuation (#4673)

  • Removed unneeded entries from the MyPy exception list (#4690)

  • Automated tests now pass on Windows as part of our continuous integration system (#4458)

  • Automated tests now pass on the latest PyPy version for supported Python versions in our continuous integration system (#4504)

Scrapy 2.2.1 (2020-07-17)

  • The startproject command no longer makes unintended changes to the permissions of files in the destination folder, such as removing execution permissions (#4662, #4666)

Scrapy 2.2.0 (2020-06-24)

Highlights:

Backward-incompatible changes

  • Support for Python 3.5.0 and 3.5.1 has been dropped; Scrapy now refuses to run with a Python version lower than 3.5.2, which introduced typing.Type (#4615)

Deprecations

New features

Bug fixes

Documentation

Quality assurance

Scrapy 2.1.0 (2020-04-24)

Highlights:

Backward-incompatible changes

  • AssertionError exceptions triggered by assert statements have been replaced by new exception types, to support running Python in optimized mode (see -O) without changing Scrapy’s behavior in any unexpected ways.

    If you catch an AssertionError exception from Scrapy, update your code to catch the corresponding new exception.

    (#4440)

Deprecation removals

  • The LOG_UNSERIALIZABLE_REQUESTS setting is no longer supported, use SCHEDULER_DEBUG instead (#4385)

  • The REDIRECT_MAX_METAREFRESH_DELAY setting is no longer supported, use METAREFRESH_MAXDELAY instead (#4385)

  • The ChunkedTransferMiddleware middleware has been removed, including the entire scrapy.downloadermiddlewares.chunked module; chunked transfers work out of the box (#4431)

  • The spiders property has been removed from Crawler, use CrawlerRunner.spider_loader or instantiate SPIDER_LOADER_CLASS with your settings instead (#4398)

  • The MultiValueDict, MultiValueDictKeyError, and SiteNode classes have been removed from scrapy.utils.datatypes (#4400)

Deprecations

  • The FEED_FORMAT and FEED_URI settings have been deprecated in favor of the new FEEDS setting (#1336, #3858, #4507)

New features

  • A new setting, FEEDS, allows configuring multiple output feeds with different settings each (#1336, #3858, #4507)

  • The crawl and runspider commands now support multiple -o parameters (#1336, #3858, #4507)

  • The crawl and runspider commands now support specifying an output format by appending :<format> to the output file (#1336, #3858, #4507)

  • The new Response.ip_address attribute gives access to the IP address that originated a response (#3903, #3940)

  • A warning is now issued when a value in allowed_domains includes a port (#50, #3198, #4413)

  • Zsh completion now excludes used option aliases from the completion list (#4438)

Bug fixes

  • Request serialization no longer breaks for callbacks that are spider attributes which are assigned a function with a different name (#4500)

  • None values in allowed_domains no longer cause a TypeError exception (#4410)

  • Zsh completion no longer allows options after arguments (#4438)

  • zope.interface 5.0.0 and later versions are now supported (#4447, #4448)

  • Spider.make_requests_from_url, deprecated in Scrapy 1.4.0, now issues a warning when used (#4412)

Documentation

  • Improved the documentation about signals that allow their handlers to return a Deferred (#4295, #4390)

  • Our PyPI entry now includes links for our documentation, our source code repository and our issue tracker (#4456)

  • Covered the curl2scrapy service in the documentation (#4206, #4455)

  • Removed references to the Guppy library, which only works in Python 2 (#4285, #4343)

  • Extended use of InterSphinx to link to Python 3 documentation (#4444, #4445)

  • Added support for Sphinx 3.0 and later (#4475, #4480, #4496, #4503)

Quality assurance

Scrapy 2.0.1 (2020-03-18)

  • Response.follow_all now supports an empty URL iterable as input (#4408, #4420)

  • Removed top-level reactor imports to prevent errors about the wrong Twisted reactor being installed when setting a different Twisted reactor using TWISTED_REACTOR (#4401, #4406)

  • Fixed tests (#4422)

Scrapy 2.0.0 (2020-03-03)

Highlights:

Backward-incompatible changes

Deprecation removals

  • The Scrapy shell no longer provides a sel proxy object, use response.selector instead (#4347)

  • LevelDB support has been removed (#4112)

  • The following functions have been removed from scrapy.utils.python: isbinarytext, is_writable, setattr_default, stringify_dict (#4362)

Deprecations

  • Using environment variables prefixed with SCRAPY_ to override settings is deprecated (#4300, #4374, #4375)

  • scrapy.linkextractors.FilteringLinkExtractor is deprecated, use scrapy.linkextractors.LinkExtractor instead (#4045)

  • The noconnect query string argument of proxy URLs is deprecated and should be removed from proxy URLs (#4198)

  • The next method of scrapy.utils.python.MutableChain is deprecated, use the global next() function or MutableChain.__next__ instead (#4153)

New features

Bug fixes

  • The crawl command now also exits with exit code 1 when an exception happens before the crawling starts (#4175, #4207)

  • LinkExtractor.extract_links no longer re-encodes the query string or URLs from non-UTF-8 responses in UTF-8 (#998, #1403, #1949, #4321)

  • The first spider middleware (see SPIDER_MIDDLEWARES) now also processes exceptions raised from callbacks that are generators (#4260, #4272)

  • Redirects to URLs starting with 3 slashes (///) are now supported (#4032, #4042)

  • Request no longer accepts strings as url simply because they have a colon (#2552, #4094)

  • The correct encoding is now used for attach names in MailSender (#4229, #4239)

  • RFPDupeFilter, the default DUPEFILTER_CLASS, no longer writes an extra \r character on each line in Windows, which made the size of the requests.seen file unnecessarily large on that platform (#4283)

  • Z shell auto-completion now looks for .html files, not .http files, and covers the -h command-line switch (#4122, #4291)

  • Adding items to a scrapy.utils.datatypes.LocalCache object without a limit defined no longer raises a TypeError exception (#4123)

  • Fixed a typo in the message of the ValueError exception raised when scrapy.utils.misc.create_instance() gets both settings and crawler set to None (#4128)

Documentation

Quality assurance

Changes to scheduler queue classes

The following changes may impact any custom queue classes of all types:

  • The push method no longer receives a second positional parameter containing request.priority * -1. If you need that value, get it from the first positional parameter, request, instead, or use the new priority() method in scrapy.core.scheduler.ScrapyPriorityQueue subclasses.

The following changes may impact custom priority queue classes:

  • In the __init__ method or the from_crawler or from_settings class methods:

    • The parameter that used to contain a factory function, qfactory, is now passed as a keyword parameter named downstream_queue_cls.

    • A new keyword parameter has been added: key. It is a string that is always an empty string for memory queues and indicates the JOBDIR value for disk queues.

    • The parameter for disk queues that contains data from the previous crawl, startprios or slot_startprios, is now passed as a keyword parameter named startprios.

    • The serialize parameter is no longer passed. The disk queue class must take care of request serialization on its own before writing to disk, using the request_to_dict() and request_from_dict() functions from the scrapy.utils.reqser module.

The following changes may impact custom disk and memory queue classes:

  • The signature of the __init__ method is now __init__(self, crawler, key).

The following changes affect specifically the ScrapyPriorityQueue and DownloaderAwarePriorityQueue classes from scrapy.core.scheduler and may affect subclasses:

  • In the __init__ method, most of the changes described above apply.

    __init__ may still receive all parameters as positional parameters, however:

    • downstream_queue_cls, which replaced qfactory, must be instantiated differently.

      qfactory was instantiated with a priority value (integer).

      Instances of downstream_queue_cls should be created using the new ScrapyPriorityQueue.qfactory or DownloaderAwarePriorityQueue.pqfactory methods.

    • The new key parameter displaced the startprios parameter 1 position to the right.

  • The following class attributes have been added:

    • crawler

    • downstream_queue_cls (details above)

    • key (details above)

  • The serialize attribute has been removed (details above)

The following changes affect specifically the ScrapyPriorityQueue class and may affect subclasses:

  • A new priority() method has been added which, given a request, returns request.priority * -1.

    It is used in push() to make up for the removal of its priority parameter.

  • The spider attribute has been removed. Use crawler.spider instead.

The following changes affect specifically the DownloaderAwarePriorityQueue class and may affect subclasses:

  • A new pqueues attribute offers a mapping of downloader slot names to the corresponding instances of downstream_queue_cls.

(#3884)