# Scrapy 0.x release notes

## Scrapy 0.24.6 (2015-04-20)

- encode invalid xpath with unicode_escape under PY2 ([commit 07cb3e5](https://github.com/scrapy/scrapy/commit/07cb3e5))
- fix IPython shell scope issue and load IPython user config ([commit 2c8e573](https://github.com/scrapy/scrapy/commit/2c8e573))
- Fix small typo in the docs ([commit d694019](https://github.com/scrapy/scrapy/commit/d694019))
- Fix small typo ([commit f92fa83](https://github.com/scrapy/scrapy/commit/f92fa83))
- Converted sel.xpath() calls to response.xpath() in Extracting the data ([commit c2c6d15](https://github.com/scrapy/scrapy/commit/c2c6d15))

## Scrapy 0.24.5 (2015-02-25)

- Support new \_getEndpoint Agent signatures on Twisted 15.0.0 ([commit 540b9bc](https://github.com/scrapy/scrapy/commit/540b9bc))
- DOC a couple more references are fixed ([commit b4c454b](https://github.com/scrapy/scrapy/commit/b4c454b))
- DOC fix a reference ([commit e3c1260](https://github.com/scrapy/scrapy/commit/e3c1260))
- t.i.b.ThreadedResolver is now a new-style class ([commit 9e13f42](https://github.com/scrapy/scrapy/commit/9e13f42))
- S3DownloadHandler: fix auth for requests with quoted paths/query params ([commit cdb9a0b](https://github.com/scrapy/scrapy/commit/cdb9a0b))
- fixed the variable types in mailsender documentation ([commit bb3a848](https://github.com/scrapy/scrapy/commit/bb3a848))
- Reset items_scraped instead of item_count ([commit edb07a4](https://github.com/scrapy/scrapy/commit/edb07a4))
- Tentative attention message about what document to read for contributions ([commit 7ee6f7a](https://github.com/scrapy/scrapy/commit/7ee6f7a))
- mitmproxy 0.10.1 needs netlib 0.10.1 too ([commit 874fcdd](https://github.com/scrapy/scrapy/commit/874fcdd))
- pin mitmproxy 0.10.1 as >0.11 does not work with tests ([commit c6b21f0](https://github.com/scrapy/scrapy/commit/c6b21f0))
- Test the parse command locally instead of against an external url ([commit c3a6628](https://github.com/scrapy/scrapy/commit/c3a6628))
- Patches Twisted issue while closing the connection pool on HTTPDownloadHandler ([commit d0bf957](https://github.com/scrapy/scrapy/commit/d0bf957))
- Updates documentation on dynamic item classes. ([commit eeb589a](https://github.com/scrapy/scrapy/commit/eeb589a))
- Merge pull request #943 from Lazar-T/patch-3 ([commit 5fdab02](https://github.com/scrapy/scrapy/commit/5fdab02))
- typo ([commit b0ae199](https://github.com/scrapy/scrapy/commit/b0ae199))
- pywin32 is required by Twisted. closes #937 ([commit 5cb0cfb](https://github.com/scrapy/scrapy/commit/5cb0cfb))
- Update install.rst ([commit 781286b](https://github.com/scrapy/scrapy/commit/781286b))
- Merge pull request #928 from Lazar-T/patch-1 ([commit b415d04](https://github.com/scrapy/scrapy/commit/b415d04))
- comma instead of fullstop ([commit 627b9ba](https://github.com/scrapy/scrapy/commit/627b9ba))
- Merge pull request #885 from jsma/patch-1 ([commit de909ad](https://github.com/scrapy/scrapy/commit/de909ad))
- Update request-response.rst ([commit 3f3263d](https://github.com/scrapy/scrapy/commit/3f3263d))
- SgmlLinkExtractor - fix for parsing <area> tag with Unicode present ([commit 49b40f0](https://github.com/scrapy/scrapy/commit/49b40f0))

## Scrapy 0.24.4 (2014-08-09)

- pem file is used by mockserver and required by scrapy bench ([commit 5eddc68b63](https://github.com/scrapy/scrapy/commit/5eddc68b63))
- scrapy bench needs scrapy.tests\* ([commit d6cb999](https://github.com/scrapy/scrapy/commit/d6cb999))

## Scrapy 0.24.3 (2014-08-09)

- no need to waste travis-ci time on py3 for 0.24 ([commit 8e080c1](https://github.com/scrapy/scrapy/commit/8e080c1))
- Update installation docs ([commit 1d0c096](https://github.com/scrapy/scrapy/commit/1d0c096))
- There is a trove classifier for Scrapy framework! ([commit 4c701d7](https://github.com/scrapy/scrapy/commit/4c701d7))
- update other places where w3lib version is mentioned ([commit d109c13](https://github.com/scrapy/scrapy/commit/d109c13))
- Update w3lib requirement to 1.8.0 ([commit 39d2ce5](https://github.com/scrapy/scrapy/commit/39d2ce5))
- Use w3lib.html.replace_entities() (remove_entities() is deprecated) ([commit 180d3ad](https://github.com/scrapy/scrapy/commit/180d3ad))
- set zip_safe=False ([commit a51ee8b](https://github.com/scrapy/scrapy/commit/a51ee8b))
- do not ship tests package ([commit ee3b371](https://github.com/scrapy/scrapy/commit/ee3b371))
- scrapy.bat is not needed anymore ([commit c3861cf](https://github.com/scrapy/scrapy/commit/c3861cf))
- Modernize setup.py ([commit 362e322](https://github.com/scrapy/scrapy/commit/362e322))
- headers can not handle non-string values ([commit 94a5c65](https://github.com/scrapy/scrapy/commit/94a5c65))
- fix ftp test cases ([commit a274a7f](https://github.com/scrapy/scrapy/commit/a274a7f))
- The sum up of travis-ci builds are taking like 50min to complete ([commit ae1e2cc](https://github.com/scrapy/scrapy/commit/ae1e2cc))
- Update shell.rst typo ([commit e49c96a](https://github.com/scrapy/scrapy/commit/e49c96a))
- removes weird indentation in the shell results ([commit 1ca489d](https://github.com/scrapy/scrapy/commit/1ca489d))
- improved explanations, clarified blog post as source, added link for XPath string functions in the spec ([commit 65c8f05](https://github.com/scrapy/scrapy/commit/65c8f05))
- renamed UserTimeoutError and ServerTimeouterror #583 ([commit 037f6ab](https://github.com/scrapy/scrapy/commit/037f6ab))
- adding some xpath tips to selectors docs ([commit 2d103e0](https://github.com/scrapy/scrapy/commit/2d103e0))
- fix tests to account for [https://github.com/scrapy/w3lib/pull/23](https://github.com/scrapy/w3lib/pull/23) ([commit f8d366a](https://github.com/scrapy/scrapy/commit/f8d366a))
- get_func_args maximum recursion fix #728 ([commit 81344ea](https://github.com/scrapy/scrapy/commit/81344ea))
- Updated input/output processor example according to #560. ([commit f7c4ea8](https://github.com/scrapy/scrapy/commit/f7c4ea8))
- Fixed Python syntax in tutorial. ([commit db59ed9](https://github.com/scrapy/scrapy/commit/db59ed9))
- Add test case for tunneling proxy ([commit f090260](https://github.com/scrapy/scrapy/commit/f090260))
- Bugfix for leaking Proxy-Authorization header to remote host when using tunneling ([commit d8793af](https://github.com/scrapy/scrapy/commit/d8793af))
- Extract links from XHTML documents with MIME-Type “application/xml” ([commit ed1f376](https://github.com/scrapy/scrapy/commit/ed1f376))
- Merge pull request #793 from roysc/patch-1 ([commit 91a1106](https://github.com/scrapy/scrapy/commit/91a1106))
- Fix typo in commands.rst ([commit 743e1e2](https://github.com/scrapy/scrapy/commit/743e1e2))
- better testcase for settings.overrides.setdefault ([commit e22daaf](https://github.com/scrapy/scrapy/commit/e22daaf))
- Using CRLF as line marker according to http 1.1 definition ([commit 5ec430b](https://github.com/scrapy/scrapy/commit/5ec430b))

## Scrapy 0.24.2 (2014-07-08)

- Use a mutable mapping to proxy deprecated settings.overrides and settings.defaults attribute ([commit e5e8133](https://github.com/scrapy/scrapy/commit/e5e8133))
- there is not support for python3 yet ([commit 3cd6146](https://github.com/scrapy/scrapy/commit/3cd6146))
- Update python compatible version set to Debian packages ([commit fa5d76b](https://github.com/scrapy/scrapy/commit/fa5d76b))
- DOC fix formatting in release notes ([commit c6a9e20](https://github.com/scrapy/scrapy/commit/c6a9e20))

## Scrapy 0.24.1 (2014-06-27)

- Fix deprecated CrawlerSettings and increase backward compatibility with
  .defaults attribute ([commit 8e3f20a](https://github.com/scrapy/scrapy/commit/8e3f20a))

## Scrapy 0.24.0 (2014-06-26)

### Enhancements

- Improve Scrapy top-level namespace ([#494](https://github.com/scrapy/scrapy/issues/494), [#684](https://github.com/scrapy/scrapy/issues/684))
- Add selector shortcuts to responses ([#554](https://github.com/scrapy/scrapy/issues/554), [#690](https://github.com/scrapy/scrapy/issues/690))
- Add new lxml based LinkExtractor to replace unmaintained SgmlLinkExtractor
  ([#559](https://github.com/scrapy/scrapy/issues/559), [#761](https://github.com/scrapy/scrapy/issues/761), [#763](https://github.com/scrapy/scrapy/issues/763))
- Cleanup settings API - part of per-spider settings **GSoC project** ([#737](https://github.com/scrapy/scrapy/issues/737))
- Add UTF8 encoding header to templates ([#688](https://github.com/scrapy/scrapy/issues/688), [#762](https://github.com/scrapy/scrapy/issues/762))
- Telnet console now binds to 127.0.0.1 by default ([#699](https://github.com/scrapy/scrapy/issues/699))
- Update Debian/Ubuntu install instructions ([#509](https://github.com/scrapy/scrapy/issues/509), [#549](https://github.com/scrapy/scrapy/issues/549))
- Disable smart strings in lxml XPath evaluations ([#535](https://github.com/scrapy/scrapy/issues/535))
- Restore filesystem based cache as default for http
  cache middleware ([#541](https://github.com/scrapy/scrapy/issues/541), [#500](https://github.com/scrapy/scrapy/issues/500), [#571](https://github.com/scrapy/scrapy/issues/571))
- Expose current crawler in Scrapy shell ([#557](https://github.com/scrapy/scrapy/issues/557))
- Improve testsuite comparing CSV and XML exporters ([#570](https://github.com/scrapy/scrapy/issues/570))
- New `offsite/filtered` and `offsite/domains` stats ([#566](https://github.com/scrapy/scrapy/issues/566))
- Support process_links as generator in CrawlSpider ([#555](https://github.com/scrapy/scrapy/issues/555))
- Verbose logging and new stats counters for DupeFilter ([#553](https://github.com/scrapy/scrapy/issues/553))
- Add a mimetype parameter to `MailSender.send()` ([#602](https://github.com/scrapy/scrapy/issues/602))
- Generalize file pipeline log messages ([#622](https://github.com/scrapy/scrapy/issues/622))
- Replace unencodeable codepoints with html entities in SGMLLinkExtractor ([#565](https://github.com/scrapy/scrapy/issues/565))
- Converted SEP documents to rst format ([#629](https://github.com/scrapy/scrapy/issues/629), [#630](https://github.com/scrapy/scrapy/issues/630),
  [#638](https://github.com/scrapy/scrapy/issues/638), [#632](https://github.com/scrapy/scrapy/issues/632), [#636](https://github.com/scrapy/scrapy/issues/636), [#640](https://github.com/scrapy/scrapy/issues/640), [#635](https://github.com/scrapy/scrapy/issues/635),
  [#634](https://github.com/scrapy/scrapy/issues/634), [#639](https://github.com/scrapy/scrapy/issues/639), [#637](https://github.com/scrapy/scrapy/issues/637), [#631](https://github.com/scrapy/scrapy/issues/631), [#633](https://github.com/scrapy/scrapy/issues/633),
  [#641](https://github.com/scrapy/scrapy/issues/641), [#642](https://github.com/scrapy/scrapy/issues/642))
- Tests and docs for clickdata’s nr index in FormRequest ([#646](https://github.com/scrapy/scrapy/issues/646), [#645](https://github.com/scrapy/scrapy/issues/645))
- Allow to disable a downloader handler just like any other component ([#650](https://github.com/scrapy/scrapy/issues/650))
- Log when a request is discarded after too many redirections ([#654](https://github.com/scrapy/scrapy/issues/654))
- Log error responses if they are not handled by spider callbacks
  ([#612](https://github.com/scrapy/scrapy/issues/612), [#656](https://github.com/scrapy/scrapy/issues/656))
- Add content-type check to http compression mw ([#193](https://github.com/scrapy/scrapy/issues/193), [#660](https://github.com/scrapy/scrapy/issues/660))
- Run pypy tests using latest pypi from ppa ([#674](https://github.com/scrapy/scrapy/issues/674))
- Run test suite using pytest instead of trial ([#679](https://github.com/scrapy/scrapy/issues/679))
- Build docs and check for dead links in tox environment ([#687](https://github.com/scrapy/scrapy/issues/687))
- Make scrapy.version_info a tuple of integers ([#681](https://github.com/scrapy/scrapy/issues/681), [#692](https://github.com/scrapy/scrapy/issues/692))
- Infer exporter’s output format from filename extensions
  ([#546](https://github.com/scrapy/scrapy/issues/546), [#659](https://github.com/scrapy/scrapy/issues/659), [#760](https://github.com/scrapy/scrapy/issues/760))
- Support case-insensitive domains in `url_is_from_any_domain()` ([#693](https://github.com/scrapy/scrapy/issues/693))
- Remove pep8 warnings in project and spider templates ([#698](https://github.com/scrapy/scrapy/issues/698))
- Tests and docs for `request_fingerprint` function ([#597](https://github.com/scrapy/scrapy/issues/597))
- Update SEP-19 for GSoC project `per-spider settings` ([#705](https://github.com/scrapy/scrapy/issues/705))
- Set exit code to non-zero when contracts fails ([#727](https://github.com/scrapy/scrapy/issues/727))
- Add a setting to control what class is instantiated as Downloader component
  ([#738](https://github.com/scrapy/scrapy/issues/738))
- Pass response in `item_dropped` signal ([#724](https://github.com/scrapy/scrapy/issues/724))
- Improve `scrapy check` contracts command ([#733](https://github.com/scrapy/scrapy/issues/733), [#752](https://github.com/scrapy/scrapy/issues/752))
- Document `spider.closed()` shortcut ([#719](https://github.com/scrapy/scrapy/issues/719))
- Document `request_scheduled` signal ([#746](https://github.com/scrapy/scrapy/issues/746))
- Add a note about reporting security issues ([#697](https://github.com/scrapy/scrapy/issues/697))
- Add LevelDB http cache storage backend ([#626](https://github.com/scrapy/scrapy/issues/626), [#500](https://github.com/scrapy/scrapy/issues/500))
- Sort spider list output of `scrapy list` command ([#742](https://github.com/scrapy/scrapy/issues/742))
- Multiple documentation enhancements and fixes
  ([#575](https://github.com/scrapy/scrapy/issues/575), [#587](https://github.com/scrapy/scrapy/issues/587), [#590](https://github.com/scrapy/scrapy/issues/590), [#596](https://github.com/scrapy/scrapy/issues/596), [#610](https://github.com/scrapy/scrapy/issues/610),
  [#617](https://github.com/scrapy/scrapy/issues/617), [#618](https://github.com/scrapy/scrapy/issues/618), [#627](https://github.com/scrapy/scrapy/issues/627), [#613](https://github.com/scrapy/scrapy/issues/613), [#643](https://github.com/scrapy/scrapy/issues/643),
  [#654](https://github.com/scrapy/scrapy/issues/654), [#675](https://github.com/scrapy/scrapy/issues/675), [#663](https://github.com/scrapy/scrapy/issues/663), [#711](https://github.com/scrapy/scrapy/issues/711), [#714](https://github.com/scrapy/scrapy/issues/714))

### Bugfixes

- Encode unicode URL value when creating Links in RegexLinkExtractor ([#561](https://github.com/scrapy/scrapy/issues/561))
- Ignore None values in ItemLoader processors ([#556](https://github.com/scrapy/scrapy/issues/556))
- Fix link text when there is an inner tag in SGMLLinkExtractor and
  HtmlParserLinkExtractor ([#485](https://github.com/scrapy/scrapy/issues/485), [#574](https://github.com/scrapy/scrapy/issues/574))
- Fix wrong checks on subclassing of deprecated classes
  ([#581](https://github.com/scrapy/scrapy/issues/581), [#584](https://github.com/scrapy/scrapy/issues/584))
- Handle errors caused by inspect.stack() failures ([#582](https://github.com/scrapy/scrapy/issues/582))
- Fix a reference to unexistent engine attribute ([#593](https://github.com/scrapy/scrapy/issues/593), [#594](https://github.com/scrapy/scrapy/issues/594))
- Fix dynamic itemclass example usage of type() ([#603](https://github.com/scrapy/scrapy/issues/603))
- Use lucasdemarchi/codespell to fix typos ([#628](https://github.com/scrapy/scrapy/issues/628))
- Fix default value of attrs argument in SgmlLinkExtractor to be tuple ([#661](https://github.com/scrapy/scrapy/issues/661))
- Fix XXE flaw in sitemap reader ([#676](https://github.com/scrapy/scrapy/issues/676))
- Fix engine to support filtered start requests ([#707](https://github.com/scrapy/scrapy/issues/707))
- Fix offsite middleware case on urls with no hostnames ([#745](https://github.com/scrapy/scrapy/issues/745))
- Testsuite doesn’t require PIL anymore ([#585](https://github.com/scrapy/scrapy/issues/585))

## Scrapy 0.22.2 (2014-02-14)

- fix a reference to unexistent engine.slots. closes #593 ([commit 13c099a](https://github.com/scrapy/scrapy/commit/13c099a))
- downloaderMW doc typo (spiderMW doc copy remnant) ([commit 8ae11bf](https://github.com/scrapy/scrapy/commit/8ae11bf))
- Correct typos ([commit 1346037](https://github.com/scrapy/scrapy/commit/1346037))

## Scrapy 0.22.1 (2014-02-08)

- localhost666 can resolve under certain circumstances ([commit 2ec2279](https://github.com/scrapy/scrapy/commit/2ec2279))
- test inspect.stack failure ([commit cc3eda3](https://github.com/scrapy/scrapy/commit/cc3eda3))
- Handle cases when inspect.stack() fails ([commit 8cb44f9](https://github.com/scrapy/scrapy/commit/8cb44f9))
- Fix wrong checks on subclassing of deprecated classes. closes #581 ([commit 46d98d6](https://github.com/scrapy/scrapy/commit/46d98d6))
- Docs: 4-space indent for final spider example ([commit 13846de](https://github.com/scrapy/scrapy/commit/13846de))
- Fix HtmlParserLinkExtractor and tests after #485 merge ([commit 368a946](https://github.com/scrapy/scrapy/commit/368a946))
- BaseSgmlLinkExtractor: Fixed the missing space when the link has an inner tag ([commit b566388](https://github.com/scrapy/scrapy/commit/b566388))
- BaseSgmlLinkExtractor: Added unit test of a link with an inner tag ([commit c1cb418](https://github.com/scrapy/scrapy/commit/c1cb418))
- BaseSgmlLinkExtractor: Fixed unknown_endtag() so that it only set current_link=None when the end tag match the opening tag ([commit 7e4d627](https://github.com/scrapy/scrapy/commit/7e4d627))
- Fix tests for Travis-CI build ([commit 76c7e20](https://github.com/scrapy/scrapy/commit/76c7e20))
- replace unencodeable codepoints with html entities. fixes #562 and #285 ([commit 5f87b17](https://github.com/scrapy/scrapy/commit/5f87b17))
- RegexLinkExtractor: encode URL unicode value when creating Links ([commit d0ee545](https://github.com/scrapy/scrapy/commit/d0ee545))
- Updated the tutorial crawl output with latest output. ([commit 8da65de](https://github.com/scrapy/scrapy/commit/8da65de))
- Updated shell docs with the crawler reference and fixed the actual shell output. ([commit 875b9ab](https://github.com/scrapy/scrapy/commit/875b9ab))
- PEP8 minor edits. ([commit f89efaf](https://github.com/scrapy/scrapy/commit/f89efaf))
- Expose current crawler in the Scrapy shell. ([commit 5349cec](https://github.com/scrapy/scrapy/commit/5349cec))
- Unused re import and PEP8 minor edits. ([commit 387f414](https://github.com/scrapy/scrapy/commit/387f414))
- Ignore None’s values when using the ItemLoader. ([commit 0632546](https://github.com/scrapy/scrapy/commit/0632546))
- DOC Fixed HTTPCACHE_STORAGE typo in the default value which is now Filesystem instead Dbm. ([commit cde9a8c](https://github.com/scrapy/scrapy/commit/cde9a8c))
- show Ubuntu setup instructions as literal code ([commit fb5c9c5](https://github.com/scrapy/scrapy/commit/fb5c9c5))
- Update Ubuntu installation instructions ([commit 70fb105](https://github.com/scrapy/scrapy/commit/70fb105))
- Merge pull request #550 from stray-leone/patch-1 ([commit 6f70b6a](https://github.com/scrapy/scrapy/commit/6f70b6a))
- modify the version of Scrapy Ubuntu package ([commit 725900d](https://github.com/scrapy/scrapy/commit/725900d))
- fix 0.22.0 release date ([commit af0219a](https://github.com/scrapy/scrapy/commit/af0219a))
- fix typos in news.rst and remove (not released yet) header ([commit b7f58f4](https://github.com/scrapy/scrapy/commit/b7f58f4))

## Scrapy 0.22.0 (2014-01-17)

### Enhancements

- [**Backward incompatible**] Switched HTTPCacheMiddleware backend to filesystem ([#541](https://github.com/scrapy/scrapy/issues/541))
  To restore old backend set `HTTPCACHE_STORAGE` to `scrapy.contrib.httpcache.DbmCacheStorage`
- Proxy https:// urls using CONNECT method ([#392](https://github.com/scrapy/scrapy/issues/392), [#397](https://github.com/scrapy/scrapy/issues/397))
- Add a middleware to crawl ajax crawlable pages as defined by google ([#343](https://github.com/scrapy/scrapy/issues/343))
- Rename scrapy.spider.BaseSpider to scrapy.spider.Spider ([#510](https://github.com/scrapy/scrapy/issues/510), [#519](https://github.com/scrapy/scrapy/issues/519))
- Selectors register EXSLT namespaces by default ([#472](https://github.com/scrapy/scrapy/issues/472))
- Unify item loaders similar to selectors renaming ([#461](https://github.com/scrapy/scrapy/issues/461))
- Make `RFPDupeFilter` class easily subclassable ([#533](https://github.com/scrapy/scrapy/issues/533))
- Improve test coverage and forthcoming Python 3 support ([#525](https://github.com/scrapy/scrapy/issues/525))
- Promote startup info on settings and middleware to INFO level ([#520](https://github.com/scrapy/scrapy/issues/520))
- Support partials in `get_func_args` util ([#506](https://github.com/scrapy/scrapy/issues/506), [#504](https://github.com/scrapy/scrapy/issues/504))
- Allow running individual tests via tox ([#503](https://github.com/scrapy/scrapy/issues/503))
- Update extensions ignored by link extractors ([#498](https://github.com/scrapy/scrapy/issues/498))
- Add middleware methods to get files/images/thumbs paths ([#490](https://github.com/scrapy/scrapy/issues/490))
- Improve offsite middleware tests ([#478](https://github.com/scrapy/scrapy/issues/478))
- Add a way to skip default Referer header set by RefererMiddleware ([#475](https://github.com/scrapy/scrapy/issues/475))
- Do not send `x-gzip` in default `Accept-Encoding` header ([#469](https://github.com/scrapy/scrapy/issues/469))
- Support defining http error handling using settings ([#466](https://github.com/scrapy/scrapy/issues/466))
- Use modern python idioms wherever you find legacies ([#497](https://github.com/scrapy/scrapy/issues/497))
- Improve and correct documentation
  ([#527](https://github.com/scrapy/scrapy/issues/527), [#524](https://github.com/scrapy/scrapy/issues/524), [#521](https://github.com/scrapy/scrapy/issues/521), [#517](https://github.com/scrapy/scrapy/issues/517), [#512](https://github.com/scrapy/scrapy/issues/512), [#505](https://github.com/scrapy/scrapy/issues/505),
  [#502](https://github.com/scrapy/scrapy/issues/502), [#489](https://github.com/scrapy/scrapy/issues/489), [#465](https://github.com/scrapy/scrapy/issues/465), [#460](https://github.com/scrapy/scrapy/issues/460), [#425](https://github.com/scrapy/scrapy/issues/425), [#536](https://github.com/scrapy/scrapy/issues/536))

### Fixes

- Update Selector class imports in CrawlSpider template ([#484](https://github.com/scrapy/scrapy/issues/484))
- Fix unexistent reference to `engine.slots` ([#464](https://github.com/scrapy/scrapy/issues/464))
- Do not try to call `body_as_unicode()` on a non-TextResponse instance ([#462](https://github.com/scrapy/scrapy/issues/462))
- Warn when subclassing XPathItemLoader, previously it only warned on
  instantiation. ([#523](https://github.com/scrapy/scrapy/issues/523))
- Warn when subclassing XPathSelector, previously it only warned on
  instantiation. ([#537](https://github.com/scrapy/scrapy/issues/537))
- Multiple fixes to memory stats ([#531](https://github.com/scrapy/scrapy/issues/531), [#530](https://github.com/scrapy/scrapy/issues/530), [#529](https://github.com/scrapy/scrapy/issues/529))
- Fix overriding url in `FormRequest.from_response()` ([#507](https://github.com/scrapy/scrapy/issues/507))
- Fix tests runner under pip 1.5 ([#513](https://github.com/scrapy/scrapy/issues/513))
- Fix logging error when spider name is unicode ([#479](https://github.com/scrapy/scrapy/issues/479))

## Scrapy 0.20.2 (2013-12-09)

- Update CrawlSpider Template with Selector changes ([commit 6d1457d](https://github.com/scrapy/scrapy/commit/6d1457d))
- fix method name in tutorial. closes GH-480 ([commit b4fc359](https://github.com/scrapy/scrapy/commit/b4fc359)

## Scrapy 0.20.1 (2013-11-28)

- include_package_data is required to build wheels from published sources ([commit 5ba1ad5](https://github.com/scrapy/scrapy/commit/5ba1ad5))
- process_parallel was leaking the failures on its internal deferreds.  closes #458 ([commit 419a780](https://github.com/scrapy/scrapy/commit/419a780))

## Scrapy 0.20.0 (2013-11-08)

### Enhancements

- New Selector’s API including CSS selectors ([#395](https://github.com/scrapy/scrapy/issues/395) and [#426](https://github.com/scrapy/scrapy/issues/426)),
- Request/Response url/body attributes are now immutable
  (modifying them had been deprecated for a long time)
- [`ITEM_PIPELINES`](../topics/settings.md) is now defined as a dict (instead of a list)
- Sitemap spider can fetch alternate URLs ([#360](https://github.com/scrapy/scrapy/issues/360))
- `Selector.remove_namespaces()` now remove namespaces from element’s attributes. ([#416](https://github.com/scrapy/scrapy/issues/416))
- Paved the road for Python 3.3+ ([#435](https://github.com/scrapy/scrapy/issues/435), [#436](https://github.com/scrapy/scrapy/issues/436), [#431](https://github.com/scrapy/scrapy/issues/431), [#452](https://github.com/scrapy/scrapy/issues/452))
- New item exporter using native python types with nesting support ([#366](https://github.com/scrapy/scrapy/issues/366))
- Tune HTTP1.1 pool size so it matches concurrency defined by settings ([commit b43b5f575](https://github.com/scrapy/scrapy/commit/b43b5f575))
- scrapy.mail.MailSender now can connect over TLS or upgrade using STARTTLS ([#327](https://github.com/scrapy/scrapy/issues/327))
- New FilesPipeline with functionality factored out from ImagesPipeline ([#370](https://github.com/scrapy/scrapy/issues/370), [#409](https://github.com/scrapy/scrapy/issues/409))
- Recommend Pillow instead of PIL for image handling ([#317](https://github.com/scrapy/scrapy/issues/317))
- Added Debian packages for Ubuntu Quantal and Raring ([commit 86230c0](https://github.com/scrapy/scrapy/commit/86230c0))
- Mock server (used for tests) can listen for HTTPS requests ([#410](https://github.com/scrapy/scrapy/issues/410))
- Remove multi spider support from multiple core components
  ([#422](https://github.com/scrapy/scrapy/issues/422), [#421](https://github.com/scrapy/scrapy/issues/421), [#420](https://github.com/scrapy/scrapy/issues/420), [#419](https://github.com/scrapy/scrapy/issues/419), [#423](https://github.com/scrapy/scrapy/issues/423), [#418](https://github.com/scrapy/scrapy/issues/418))
- Travis-CI now tests Scrapy changes against development versions of `w3lib` and `queuelib` python packages.
- Add pypy 2.1 to continuous integration tests ([commit ecfa7431](https://github.com/scrapy/scrapy/commit/ecfa7431))
- Pylinted, pep8 and removed old-style exceptions from source ([#430](https://github.com/scrapy/scrapy/issues/430), [#432](https://github.com/scrapy/scrapy/issues/432))
- Use importlib for parametric imports ([#445](https://github.com/scrapy/scrapy/issues/445))
- Handle a regression introduced in Python 2.7.5 that affects XmlItemExporter ([#372](https://github.com/scrapy/scrapy/issues/372))
- Bugfix crawling shutdown on SIGINT ([#450](https://github.com/scrapy/scrapy/issues/450))
- Do not submit `reset` type inputs in FormRequest.from_response ([commit b326b87](https://github.com/scrapy/scrapy/commit/b326b87))
- Do not silence download errors when request errback raises an exception ([commit 684cfc0](https://github.com/scrapy/scrapy/commit/684cfc0))

### Bugfixes

- Fix tests under Django 1.6 ([commit b6bed44c](https://github.com/scrapy/scrapy/commit/b6bed44c))
- Lot of bugfixes to retry middleware under disconnections using HTTP 1.1 download handler
- Fix inconsistencies among Twisted releases ([#406](https://github.com/scrapy/scrapy/issues/406))
- Fix Scrapy shell bugs ([#418](https://github.com/scrapy/scrapy/issues/418), [#407](https://github.com/scrapy/scrapy/issues/407))
- Fix invalid variable name in setup.py ([#429](https://github.com/scrapy/scrapy/issues/429))
- Fix tutorial references ([#387](https://github.com/scrapy/scrapy/issues/387))
- Improve request-response docs ([#391](https://github.com/scrapy/scrapy/issues/391))
- Improve best practices docs ([#399](https://github.com/scrapy/scrapy/issues/399), [#400](https://github.com/scrapy/scrapy/issues/400), [#401](https://github.com/scrapy/scrapy/issues/401), [#402](https://github.com/scrapy/scrapy/issues/402))
- Improve django integration docs ([#404](https://github.com/scrapy/scrapy/issues/404))
- Document `bindaddress` request meta ([commit 37c24e01d7](https://github.com/scrapy/scrapy/commit/37c24e01d7))
- Improve `Request` class documentation ([#226](https://github.com/scrapy/scrapy/issues/226))

### Other

- Dropped Python 2.6 support ([#448](https://github.com/scrapy/scrapy/issues/448))
- Add [cssselect](https://cssselect.readthedocs.io/en/latest/index.html) python package as install dependency
- Drop libxml2 and multi selector’s backend support, [lxml](https://lxml.de/) is required from now on.
- Minimum Twisted version increased to 10.0.0, dropped Twisted 8.0 support.
- Running test suite now requires `mock` python library ([#390](https://github.com/scrapy/scrapy/issues/390))

### Thanks

Thanks to everyone who contribute to this release!

List of contributors sorted by number of commits:

```
69 Daniel Graña <dangra@...>
37 Pablo Hoffman <pablo@...>
13 Mikhail Korobov <kmike84@...>
 9 Alex Cepoi <alex.cepoi@...>
 9 alexanderlukanin13 <alexander.lukanin.13@...>
 8 Rolando Espinoza La fuente <darkrho@...>
 8 Lukasz Biedrycki <lukasz.biedrycki@...>
 6 Nicolas Ramirez <nramirez.uy@...>
 3 Paul Tremberth <paul.tremberth@...>
 2 Martin Olveyra <molveyra@...>
 2 Stefan <misc@...>
 2 Rolando Espinoza <darkrho@...>
 2 Loren Davie <loren@...>
 2 irgmedeiros <irgmedeiros@...>
 1 Stefan Koch <taikano@...>
 1 Stefan <cct@...>
 1 scraperdragon <dragon@...>
 1 Kumara Tharmalingam <ktharmal@...>
 1 Francesco Piccinno <stack.box@...>
 1 Marcos Campal <duendex@...>
 1 Dragon Dave <dragon@...>
 1 Capi Etheriel <barraponto@...>
 1 cacovsky <amarquesferraz@...>
 1 Berend Iwema <berend@...>
```

## Scrapy 0.18.4 (2013-10-10)

- IPython refuses to update the namespace. fix #396 ([commit 3d32c4f](https://github.com/scrapy/scrapy/commit/3d32c4f))
- Fix AlreadyCalledError replacing a request in shell command. closes #407 ([commit b1d8919](https://github.com/scrapy/scrapy/commit/b1d8919))
- Fix `start_requests()` laziness and early hangs ([commit 89faf52](https://github.com/scrapy/scrapy/commit/89faf52))

## Scrapy 0.18.3 (2013-10-03)

- fix regression on lazy evaluation of start requests ([commit 12693a5](https://github.com/scrapy/scrapy/commit/12693a5))
- forms: do not submit reset inputs ([commit e429f63](https://github.com/scrapy/scrapy/commit/e429f63))
- increase unittest timeouts to decrease travis false positive failures ([commit 912202e](https://github.com/scrapy/scrapy/commit/912202e))
- backport master fixes to json exporter ([commit cfc2d46](https://github.com/scrapy/scrapy/commit/cfc2d46))
- Fix permission and set umask before generating sdist tarball ([commit 06149e0](https://github.com/scrapy/scrapy/commit/06149e0))

## Scrapy 0.18.2 (2013-09-03)

- Backport `scrapy check` command fixes and backward compatible multi
  crawler process([#339](https://github.com/scrapy/scrapy/issues/339))

## Scrapy 0.18.1 (2013-08-27)

- remove extra import added by cherry picked changes ([commit d20304e](https://github.com/scrapy/scrapy/commit/d20304e))
- fix crawling tests under twisted pre 11.0.0 ([commit 1994f38](https://github.com/scrapy/scrapy/commit/1994f38))
- py26 can not format zero length fields {} ([commit abf756f](https://github.com/scrapy/scrapy/commit/abf756f))
- test PotentiaDataLoss errors on unbound responses ([commit b15470d](https://github.com/scrapy/scrapy/commit/b15470d))
- Treat responses without content-length or Transfer-Encoding as good responses ([commit c4bf324](https://github.com/scrapy/scrapy/commit/c4bf324))
- do no include ResponseFailed if http11 handler is not enabled ([commit 6cbe684](https://github.com/scrapy/scrapy/commit/6cbe684))
- New HTTP client wraps connection lost in ResponseFailed exception. fix #373 ([commit 1a20bba](https://github.com/scrapy/scrapy/commit/1a20bba))
- limit travis-ci build matrix ([commit 3b01bb8](https://github.com/scrapy/scrapy/commit/3b01bb8))
- Merge pull request #375 from peterarenot/patch-1 ([commit fa766d7](https://github.com/scrapy/scrapy/commit/fa766d7))
- Fixed so it refers to the correct folder ([commit 3283809](https://github.com/scrapy/scrapy/commit/3283809))
- added Quantal & Raring to support Ubuntu releases ([commit 1411923](https://github.com/scrapy/scrapy/commit/1411923))
- fix retry middleware which didn’t retry certain connection errors after the upgrade to http1 client, closes GH-373 ([commit bb35ed0](https://github.com/scrapy/scrapy/commit/bb35ed0))
- fix XmlItemExporter in Python 2.7.4 and 2.7.5 ([commit de3e451](https://github.com/scrapy/scrapy/commit/de3e451))
- minor updates to 0.18 release notes ([commit c45e5f1](https://github.com/scrapy/scrapy/commit/c45e5f1))
- fix contributors list format ([commit 0b60031](https://github.com/scrapy/scrapy/commit/0b60031))

## Scrapy 0.18.0 (2013-08-09)

- Lot of improvements to testsuite run using Tox, including a way to test on pypi
- Handle GET parameters for AJAX crawlable urls ([commit 3fe2a32](https://github.com/scrapy/scrapy/commit/3fe2a32))
- Use lxml recover option to parse sitemaps ([#347](https://github.com/scrapy/scrapy/issues/347))
- Bugfix cookie merging by hostname and not by netloc ([#352](https://github.com/scrapy/scrapy/issues/352))
- Support disabling `HttpCompressionMiddleware` using a flag setting ([#359](https://github.com/scrapy/scrapy/issues/359))
- Support xml namespaces using `iternodes` parser in `XMLFeedSpider` ([#12](https://github.com/scrapy/scrapy/issues/12))
- Support `dont_cache` request meta flag ([#19](https://github.com/scrapy/scrapy/issues/19))
- Bugfix `scrapy.utils.gz.gunzip` broken by changes in python 2.7.4 ([commit 4dc76e](https://github.com/scrapy/scrapy/commit/4dc76e))
- Bugfix url encoding on `SgmlLinkExtractor` ([#24](https://github.com/scrapy/scrapy/issues/24))
- Bugfix `TakeFirst` processor shouldn’t discard zero (0) value ([#59](https://github.com/scrapy/scrapy/issues/59))
- Support nested items in xml exporter ([#66](https://github.com/scrapy/scrapy/issues/66))
- Improve cookies handling performance ([#77](https://github.com/scrapy/scrapy/issues/77))
- Log dupe filtered requests once ([#105](https://github.com/scrapy/scrapy/issues/105))
- Split redirection middleware into status and meta based middlewares ([#78](https://github.com/scrapy/scrapy/issues/78))
- Use HTTP1.1 as default downloader handler ([#109](https://github.com/scrapy/scrapy/issues/109) and [#318](https://github.com/scrapy/scrapy/issues/318))
- Support xpath form selection on `FormRequest.from_response` ([#185](https://github.com/scrapy/scrapy/issues/185))
- Bugfix unicode decoding error on `SgmlLinkExtractor` ([#199](https://github.com/scrapy/scrapy/issues/199))
- Bugfix signal dispatching on pypi interpreter ([#205](https://github.com/scrapy/scrapy/issues/205))
- Improve request delay and concurrency handling ([#206](https://github.com/scrapy/scrapy/issues/206))
- Add RFC2616 cache policy to `HttpCacheMiddleware` ([#212](https://github.com/scrapy/scrapy/issues/212))
- Allow customization of messages logged by engine ([#214](https://github.com/scrapy/scrapy/issues/214))
- Multiples improvements to `DjangoItem` ([#217](https://github.com/scrapy/scrapy/issues/217), [#218](https://github.com/scrapy/scrapy/issues/218), [#221](https://github.com/scrapy/scrapy/issues/221))
- Extend Scrapy commands using setuptools entry points ([#260](https://github.com/scrapy/scrapy/issues/260))
- Allow spider `allowed_domains` value to be set/tuple ([#261](https://github.com/scrapy/scrapy/issues/261))
- Support `settings.getdict` ([#269](https://github.com/scrapy/scrapy/issues/269))
- Simplify internal `scrapy.core.scraper` slot handling ([#271](https://github.com/scrapy/scrapy/issues/271))
- Added `Item.copy` ([#290](https://github.com/scrapy/scrapy/issues/290))
- Collect idle downloader slots ([#297](https://github.com/scrapy/scrapy/issues/297))
- Add `ftp://` scheme downloader handler ([#329](https://github.com/scrapy/scrapy/issues/329))
- Added downloader benchmark webserver and spider tools [Benchmarking](../topics/benchmarking.md)
- Moved persistent (on disk) queues to a separate project ([queuelib](https://github.com/scrapy/queuelib)) which Scrapy now depends on
- Add Scrapy commands using external libraries ([#260](https://github.com/scrapy/scrapy/issues/260))
- Added `--pdb` option to `scrapy` command line tool
- Added [`XPathSelector.remove_namespaces`](../topics/selectors.md) which allows to remove all namespaces from XML documents for convenience (to work with namespace-less XPaths). Documented in [Selectors](../topics/selectors.md).
- Several improvements to spider contracts
- New default middleware named MetaRefreshMiddleware that handles meta-refresh html tag redirections,
- MetaRefreshMiddleware and RedirectMiddleware have different priorities to address #62
- added from_crawler method to spiders
- added system tests with mock server
- more improvements to macOS compatibility (thanks Alex Cepoi)
- several more cleanups to singletons and multi-spider support (thanks Nicolas Ramirez)
- support custom download slots
- added –spider option to “shell” command.
- log overridden settings when Scrapy starts

Thanks to everyone who contribute to this release. Here is a list of
contributors sorted by number of commits:

```
130 Pablo Hoffman <pablo@...>
 97 Daniel Graña <dangra@...>
 20 Nicolás Ramírez <nramirez.uy@...>
 13 Mikhail Korobov <kmike84@...>
 12 Pedro Faustino <pedrobandim@...>
 11 Steven Almeroth <sroth77@...>
  5 Rolando Espinoza La fuente <darkrho@...>
  4 Michal Danilak <mimino.coder@...>
  4 Alex Cepoi <alex.cepoi@...>
  4 Alexandr N Zamaraev (aka tonal) <tonal@...>
  3 paul <paul.tremberth@...>
  3 Martin Olveyra <molveyra@...>
  3 Jordi Llonch <llonchj@...>
  3 arijitchakraborty <myself.arijit@...>
  2 Shane Evans <shane.evans@...>
  2 joehillen <joehillen@...>
  2 Hart <HartSimha@...>
  2 Dan <ellisd23@...>
  1 Zuhao Wan <wanzuhao@...>
  1 whodatninja <blake@...>
  1 vkrest <v.krestiannykov@...>
  1 tpeng <pengtaoo@...>
  1 Tom Mortimer-Jones <tom@...>
  1 Rocio Aramberri <roschegel@...>
  1 Pedro <pedro@...>
  1 notsobad <wangxiaohugg@...>
  1 Natan L <kuyanatan.nlao@...>
  1 Mark Grey <mark.grey@...>
  1 Luan <luanpab@...>
  1 Libor Nenadál <libor.nenadal@...>
  1 Juan M Uys <opyate@...>
  1 Jonas Brunsgaard <jonas.brunsgaard@...>
  1 Ilya Baryshev <baryshev@...>
  1 Hasnain Lakhani <m.hasnain.lakhani@...>
  1 Emanuel Schorsch <emschorsch@...>
  1 Chris Tilden <chris.tilden@...>
  1 Capi Etheriel <barraponto@...>
  1 cacovsky <amarquesferraz@...>
  1 Berend Iwema <berend@...>
```

## Scrapy 0.16.5 (2013-05-30)

- obey request method when Scrapy deploy is redirected to a new endpoint ([commit 8c4fcee](https://github.com/scrapy/scrapy/commit/8c4fcee))
- fix inaccurate downloader middleware documentation. refs #280 ([commit 40667cb](https://github.com/scrapy/scrapy/commit/40667cb))
- doc: remove links to diveintopython.org, which is no longer available. closes #246 ([commit bd58bfa](https://github.com/scrapy/scrapy/commit/bd58bfa))
- Find form nodes in invalid html5 documents ([commit e3d6945](https://github.com/scrapy/scrapy/commit/e3d6945))
- Fix typo labeling attrs type bool instead of list ([commit a274276](https://github.com/scrapy/scrapy/commit/a274276))

## Scrapy 0.16.4 (2013-01-23)

- fixes spelling errors in documentation ([commit 6d2b3aa](https://github.com/scrapy/scrapy/commit/6d2b3aa))
- add doc about disabling an extension. refs #132 ([commit c90de33](https://github.com/scrapy/scrapy/commit/c90de33))
- Fixed error message formatting. log.err() doesn’t support cool formatting and when error occurred, the message was:    “ERROR: Error processing %(item)s” ([commit c16150c](https://github.com/scrapy/scrapy/commit/c16150c))
- lint and improve images pipeline error logging ([commit 56b45fc](https://github.com/scrapy/scrapy/commit/56b45fc))
- fixed doc typos ([commit 243be84](https://github.com/scrapy/scrapy/commit/243be84))
- add documentation topics: Broad Crawls & Common Practices ([commit 1fbb715](https://github.com/scrapy/scrapy/commit/1fbb715))
- fix bug in Scrapy parse command when spider is not specified explicitly. closes #209 ([commit c72e682](https://github.com/scrapy/scrapy/commit/c72e682))
- Update docs/topics/commands.rst ([commit 28eac7a](https://github.com/scrapy/scrapy/commit/28eac7a))

## Scrapy 0.16.3 (2012-12-07)

- Remove concurrency limitation when using download delays and still ensure inter-request delays are enforced ([commit 487b9b5](https://github.com/scrapy/scrapy/commit/487b9b5))
- add error details when image pipeline fails ([commit 8232569](https://github.com/scrapy/scrapy/commit/8232569))
- improve macOS compatibility ([commit 8dcf8aa](https://github.com/scrapy/scrapy/commit/8dcf8aa))
- setup.py: use README.rst to populate long_description ([commit 7b5310d](https://github.com/scrapy/scrapy/commit/7b5310d))
- doc: removed obsolete references to ClientForm ([commit 80f9bb6](https://github.com/scrapy/scrapy/commit/80f9bb6))
- correct docs for default storage backend ([commit 2aa491b](https://github.com/scrapy/scrapy/commit/2aa491b))
- doc: removed broken proxyhub link from FAQ ([commit bdf61c4](https://github.com/scrapy/scrapy/commit/bdf61c4))
- Fixed docs typo in SpiderOpenCloseLogging example ([commit 7184094](https://github.com/scrapy/scrapy/commit/7184094))

## Scrapy 0.16.2 (2012-11-09)

- Scrapy contracts: python2.6 compat ([commit a4a9199](https://github.com/scrapy/scrapy/commit/a4a9199))
- Scrapy contracts verbose option ([commit ec41673](https://github.com/scrapy/scrapy/commit/ec41673))
- proper unittest-like output for Scrapy contracts ([commit 86635e4](https://github.com/scrapy/scrapy/commit/86635e4))
- added open_in_browser to debugging doc ([commit c9b690d](https://github.com/scrapy/scrapy/commit/c9b690d))
- removed reference to global Scrapy stats from settings doc ([commit dd55067](https://github.com/scrapy/scrapy/commit/dd55067))
- Fix SpiderState bug in Windows platforms ([commit 58998f4](https://github.com/scrapy/scrapy/commit/58998f4))

## Scrapy 0.16.1 (2012-10-26)

- fixed LogStats extension, which got broken after a wrong merge before the 0.16 release ([commit 8c780fd](https://github.com/scrapy/scrapy/commit/8c780fd))
- better backward compatibility for scrapy.conf.settings ([commit 3403089](https://github.com/scrapy/scrapy/commit/3403089))
- extended documentation on how to access crawler stats from extensions ([commit c4da0b5](https://github.com/scrapy/scrapy/commit/c4da0b5))
- removed .hgtags (no longer needed now that Scrapy uses git) ([commit d52c188](https://github.com/scrapy/scrapy/commit/d52c188))
- fix dashes under rst headers ([commit fa4f7f9](https://github.com/scrapy/scrapy/commit/fa4f7f9))
- set release date for 0.16.0 in news ([commit e292246](https://github.com/scrapy/scrapy/commit/e292246))

## Scrapy 0.16.0 (2012-10-18)

Scrapy changes:

- added [Spider Contracts](../topics/contracts.md), a mechanism for testing spiders in a formal/reproducible way
- added options `-o` and `-t` to the [`runspider`](../topics/commands.md) command
- documented [AutoThrottle extension](https://docs.scrapy.org/en/latest/topics/autothrottle.md) and added to extensions installed by default. You still need to enable it with [`AUTOTHROTTLE_ENABLED`](../topics/autothrottle.md)
- major Stats Collection refactoring: removed separation of global/per-spider stats, removed stats-related signals (`stats_spider_opened`, etc). Stats are much simpler now, backward compatibility is kept on the Stats Collector API and signals.
- added a `process_start_requests()` method to spider middlewares
- dropped Signals singleton. Signals should now be accessed through the Crawler.signals attribute. See the signals documentation for more info.
- dropped Stats Collector singleton. Stats can now be accessed through the Crawler.stats attribute. See the stats collection documentation for more info.
- documented [Core API](../topics/api.md)
- `lxml` is now the default selectors backend instead of `libxml2`
- ported FormRequest.from_response() to use [lxml](https://lxml.de/) instead of [ClientForm](https://pypi.org/project/ClientForm/)
- removed modules: `scrapy.xlib.BeautifulSoup` and `scrapy.xlib.ClientForm`
- SitemapSpider: added support for sitemap urls ending in .xml and .xml.gz, even if they advertise a wrong content type ([commit 10ed28b](https://github.com/scrapy/scrapy/commit/10ed28b))
- StackTraceDump extension: also dump trackref live references ([commit fe2ce93](https://github.com/scrapy/scrapy/commit/fe2ce93))
- nested items now fully supported in JSON and JSONLines exporters
- added [`cookiejar`](../topics/cookies.md) Request meta key to support multiple cookie sessions per spider
- decoupled encoding detection code to [w3lib.encoding](https://github.com/scrapy/w3lib/blob/master/w3lib/encoding.py), and ported Scrapy code to use that module
- dropped support for Python 2.5. See [https://www.zyte.com/blog/scrapy-0-15-dropping-support-for-python-2-5/](https://www.zyte.com/blog/scrapy-0-15-dropping-support-for-python-2-5/)
- dropped support for Twisted 2.5
- added [`REFERER_ENABLED`](../topics/spider-middleware.md) setting, to control referer middleware
- changed default user agent to: `Scrapy/VERSION (+http://scrapy.org)`
- removed (undocumented) `HTMLImageLinkExtractor` class from `scrapy.contrib.linkextractors.image`
- removed per-spider settings (to be replaced by instantiating multiple crawler objects)
- `USER_AGENT` spider attribute will no longer work, use `user_agent` attribute instead
- `DOWNLOAD_TIMEOUT` spider attribute will no longer work, use `download_timeout` attribute instead
- removed `ENCODING_ALIASES` setting, as encoding auto-detection has been moved to the [w3lib](https://github.com/scrapy/w3lib) library
- promoted [DjangoItem](../topics/djangoitem.md) to main contrib
- LogFormatter method now return dicts(instead of strings) to support lazy formatting ([#164](https://github.com/scrapy/scrapy/issues/164), [commit dcef7b0](https://github.com/scrapy/scrapy/commit/dcef7b0))
- downloader handlers ([`DOWNLOAD_HANDLERS`](../topics/settings.md) setting) now receive settings as the first argument of the `__init__` method
- replaced memory usage accounting with (more portable) [resource](https://docs.python.org/2/library/resource.html) module, removed `scrapy.utils.memory` module
- removed signal: `scrapy.mail.mail_sent`
- removed `TRACK_REFS` setting, now [trackrefs](../topics/leaks.md "Debugging memory leaks with trackref") is always enabled
- DBM is now the default storage backend for HTTP cache middleware
- number of log messages (per level) are now tracked through Scrapy stats (stat name: `log_count/LEVEL`)
- number received responses are now tracked through Scrapy stats (stat name: `response_received_count`)
- removed `scrapy.log.started` attribute

## Scrapy 0.14.4 (2012-05-15)

- added precise to supported Ubuntu distros ([commit b7e46df](https://github.com/scrapy/scrapy/commit/b7e46df))
- fixed bug in json-rpc webservice reported in [https://groups.google.com/forum/#!topic/scrapy-users/qgVBmFybNAQ/discussion](https://groups.google.com/forum/#!topic/scrapy-users/qgVBmFybNAQ/discussion). also removed no longer supported ‘run’ command from extras/scrapy-ws.py ([commit 340fbdb](https://github.com/scrapy/scrapy/commit/340fbdb))
- meta tag attributes for content-type http equiv can be in any order. #123 ([commit 0cb68af](https://github.com/scrapy/scrapy/commit/0cb68af))
- replace “import Image” by more standard “from PIL import Image”. closes #88 ([commit 4d17048](https://github.com/scrapy/scrapy/commit/4d17048))
- return trial status as bin/runtests.sh exit value. #118 ([commit b7b2e7f](https://github.com/scrapy/scrapy/commit/b7b2e7f))

## Scrapy 0.14.3 (2012-04-19)

- forgot to include pydispatch license. #118 ([commit fd85f9c](https://github.com/scrapy/scrapy/commit/fd85f9c))
- include egg files used by testsuite in source distribution. #118 ([commit c897793](https://github.com/scrapy/scrapy/commit/c897793))
- update docstring in project template to avoid confusion with genspider command, which may be considered as an advanced feature. refs #107 ([commit 2548dcc](https://github.com/scrapy/scrapy/commit/2548dcc))
- added note to docs/topics/firebug.rst about google directory being shut down ([commit 668e352](https://github.com/scrapy/scrapy/commit/668e352))
- don’t discard slot when empty, just save in another dict in order to recycle if needed again. ([commit 8e9f607](https://github.com/scrapy/scrapy/commit/8e9f607))
- do not fail handling unicode xpaths in libxml2 backed selectors ([commit b830e95](https://github.com/scrapy/scrapy/commit/b830e95))
- fixed minor mistake in Request objects documentation ([commit bf3c9ee](https://github.com/scrapy/scrapy/commit/bf3c9ee))
- fixed minor defect in link extractors documentation ([commit ba14f38](https://github.com/scrapy/scrapy/commit/ba14f38))
- removed some obsolete remaining code related to sqlite support in Scrapy ([commit 0665175](https://github.com/scrapy/scrapy/commit/0665175))

## Scrapy 0.14.2 (2012-03-11)

- move buffer pointing to start of file before computing checksum. refs #92 ([commit 6a5bef2](https://github.com/scrapy/scrapy/commit/6a5bef2))
- Compute image checksum before persisting images. closes #92 ([commit 9817df1](https://github.com/scrapy/scrapy/commit/9817df1))
- remove leaking references in cached failures ([commit 673a120](https://github.com/scrapy/scrapy/commit/673a120))
- fixed bug in MemoryUsage extension: get_engine_status() takes exactly 1 argument (0 given) ([commit 11133e9](https://github.com/scrapy/scrapy/commit/11133e9))
- fixed struct.error on http compression middleware. closes #87 ([commit 1423140](https://github.com/scrapy/scrapy/commit/1423140))
- ajax crawling wasn’t expanding for unicode urls ([commit 0de3fb4](https://github.com/scrapy/scrapy/commit/0de3fb4))
- Catch `start_requests()` iterator errors. refs #83 ([commit 454a21d](https://github.com/scrapy/scrapy/commit/454a21d))
- Speed-up libxml2 XPathSelector ([commit 2fbd662](https://github.com/scrapy/scrapy/commit/2fbd662))
- updated versioning doc according to recent changes ([commit 0a070f5](https://github.com/scrapy/scrapy/commit/0a070f5))
- scrapyd: fixed documentation link ([commit 2b4e4c3](https://github.com/scrapy/scrapy/commit/2b4e4c3))
- extras/makedeb.py: no longer obtaining version from git ([commit caffe0e](https://github.com/scrapy/scrapy/commit/caffe0e))

## Scrapy 0.14.1 (2012-01-02)

- extras/makedeb.py: no longer obtaining version from git ([commit caffe0e](https://github.com/scrapy/scrapy/commit/caffe0e))
- bumped version to 0.14.1 ([commit 6cb9e1c](https://github.com/scrapy/scrapy/commit/6cb9e1c))
- fixed reference to tutorial directory ([commit 4b86bd6](https://github.com/scrapy/scrapy/commit/4b86bd6))
- doc: removed duplicated callback argument from Request.replace() ([commit 1aeccdd](https://github.com/scrapy/scrapy/commit/1aeccdd))
- fixed formatting of scrapyd doc ([commit 8bf19e6](https://github.com/scrapy/scrapy/commit/8bf19e6))
- Dump stacks for all running threads and fix engine status dumped by StackTraceDump extension ([commit 14a8e6e](https://github.com/scrapy/scrapy/commit/14a8e6e))
- added comment about why we disable ssl on boto images upload ([commit 5223575](https://github.com/scrapy/scrapy/commit/5223575))
- SSL handshaking hangs when doing too many parallel connections to S3 ([commit 63d583d](https://github.com/scrapy/scrapy/commit/63d583d))
- change tutorial to follow changes on dmoz site ([commit bcb3198](https://github.com/scrapy/scrapy/commit/bcb3198))
- Avoid \_disconnectedDeferred AttributeError exception in Twisted>=11.1.0 ([commit 98f3f87](https://github.com/scrapy/scrapy/commit/98f3f87))
- allow spider to set autothrottle max concurrency ([commit 175a4b5](https://github.com/scrapy/scrapy/commit/175a4b5))

## Scrapy 0.14 (2011-12-31)

### New features and settings

- Support for AJAX crawlable urls
- New persistent scheduler that stores requests on disk, allowing to suspend and resume crawls ([r2737](http://hg.scrapy.org/scrapy/changeset/2737))
- added `-o` option to `scrapy crawl`, a shortcut for dumping scraped items into a file (or standard output using `-`)
- Added support for passing custom settings to Scrapyd `schedule.json` api ([r2779](http://hg.scrapy.org/scrapy/changeset/2779), [r2783](http://hg.scrapy.org/scrapy/changeset/2783))
- New `ChunkedTransferMiddleware` (enabled by default) to support [chunked transfer encoding](https://en.wikipedia.org/wiki/Chunked_transfer_encoding) ([r2769](http://hg.scrapy.org/scrapy/changeset/2769))
- Add boto 2.0 support for S3 downloader handler ([r2763](http://hg.scrapy.org/scrapy/changeset/2763))
- Added [marshal](https://docs.python.org/2/library/marshal.html) to formats supported by feed exports ([r2744](http://hg.scrapy.org/scrapy/changeset/2744))
- In request errbacks, offending requests are now received in `failure.request` attribute ([r2738](http://hg.scrapy.org/scrapy/changeset/2738))
- Big downloader refactoring to support per domain/ip concurrency limits ([r2732](http://hg.scrapy.org/scrapy/changeset/2732))
  : - `CONCURRENT_REQUESTS_PER_SPIDER` setting has been deprecated and replaced by:
      : - [`CONCURRENT_REQUESTS`](../topics/settings.md), [`CONCURRENT_REQUESTS_PER_DOMAIN`](../topics/settings.md), `CONCURRENT_REQUESTS_PER_IP`
    - check the documentation for more details
- Added builtin caching DNS resolver ([r2728](http://hg.scrapy.org/scrapy/changeset/2728))
- Moved Amazon AWS-related components/extensions (SQS spider queue, SimpleDB stats collector) to a separate project: [scaws]([https://github.com/scrapinghub/scaws](https://github.com/scrapinghub/scaws)) ([r2706](http://hg.scrapy.org/scrapy/changeset/2706), [r2714](http://hg.scrapy.org/scrapy/changeset/2714))
- Moved spider queues to scrapyd: `scrapy.spiderqueue` -> `scrapyd.spiderqueue` ([r2708](http://hg.scrapy.org/scrapy/changeset/2708))
- Moved sqlite utils to scrapyd: `scrapy.utils.sqlite` -> `scrapyd.sqlite` ([r2781](http://hg.scrapy.org/scrapy/changeset/2781))
- Real support for returning iterators on `start_requests()` method. The iterator is now consumed during the crawl when the spider is getting idle ([r2704](http://hg.scrapy.org/scrapy/changeset/2704))
- Added [`REDIRECT_ENABLED`](../topics/downloader-middleware.md) setting to quickly enable/disable the redirect middleware ([r2697](http://hg.scrapy.org/scrapy/changeset/2697))
- Added [`RETRY_ENABLED`](../topics/downloader-middleware.md) setting to quickly enable/disable the retry middleware ([r2694](http://hg.scrapy.org/scrapy/changeset/2694))
- Added `CloseSpider` exception to manually close spiders ([r2691](http://hg.scrapy.org/scrapy/changeset/2691))
- Improved encoding detection by adding support for HTML5 meta charset declaration ([r2690](http://hg.scrapy.org/scrapy/changeset/2690))
- Refactored close spider behavior to wait for all downloads to finish and be processed by spiders, before closing the spider ([r2688](http://hg.scrapy.org/scrapy/changeset/2688))
- Added `SitemapSpider` (see documentation in Spiders page) ([r2658](http://hg.scrapy.org/scrapy/changeset/2658))
- Added `LogStats` extension for periodically logging basic stats (like crawled pages and scraped items) ([r2657](http://hg.scrapy.org/scrapy/changeset/2657))
- Make handling of gzipped responses more robust (#319, [r2643](http://hg.scrapy.org/scrapy/changeset/2643)). Now Scrapy will try and decompress as much as possible from a gzipped response, instead of failing with an `IOError`.
- Simplified !MemoryDebugger extension to use stats for dumping memory debugging info ([r2639](http://hg.scrapy.org/scrapy/changeset/2639))
- Added new command to edit spiders: `scrapy edit` ([r2636](http://hg.scrapy.org/scrapy/changeset/2636)) and `-e` flag to `genspider` command that uses it ([r2653](http://hg.scrapy.org/scrapy/changeset/2653))
- Changed default representation of items to pretty-printed dicts. ([r2631](http://hg.scrapy.org/scrapy/changeset/2631)). This improves default logging by making log more readable in the default case, for both Scraped and Dropped lines.
- Added [`spider_error`](../topics/signals.md) signal ([r2628](http://hg.scrapy.org/scrapy/changeset/2628))
- Added [`COOKIES_ENABLED`](../topics/cookies.md) setting ([r2625](http://hg.scrapy.org/scrapy/changeset/2625))
- Stats are now dumped to Scrapy log (default value of [`STATS_DUMP`](../topics/settings.md) setting has been changed to `True`). This is to make Scrapy users more aware of Scrapy stats and the data that is collected there.
- Added support for dynamically adjusting download delay and maximum concurrent requests ([r2599](http://hg.scrapy.org/scrapy/changeset/2599))
- Added new DBM HTTP cache storage backend ([r2576](http://hg.scrapy.org/scrapy/changeset/2576))
- Added `listjobs.json` API to Scrapyd ([r2571](http://hg.scrapy.org/scrapy/changeset/2571))
- `CsvItemExporter`: added `join_multivalued` parameter ([r2578](http://hg.scrapy.org/scrapy/changeset/2578))
- Added namespace support to `xmliter_lxml` ([r2552](http://hg.scrapy.org/scrapy/changeset/2552))
- Improved cookies middleware by making `COOKIES_DEBUG` nicer and documenting it ([r2579](http://hg.scrapy.org/scrapy/changeset/2579))
- Several improvements to Scrapyd and Link extractors

### Code rearranged and removed

- Merged item passed and item scraped concepts, as they have often proved confusing in the past. This means: ([r2630](http://hg.scrapy.org/scrapy/changeset/2630))
  : - original item_scraped signal was removed
    - original item_passed signal was renamed to item_scraped
    - old log lines `Scraped Item...` were removed
    - old log lines `Passed Item...` were renamed to `Scraped Item...` lines and downgraded to `DEBUG` level
- Reduced Scrapy codebase by striping part of Scrapy code into two new libraries:
  : - [w3lib](https://github.com/scrapy/w3lib) (several functions from `scrapy.utils.{http,markup,multipart,response,url}`, done in [r2584](http://hg.scrapy.org/scrapy/changeset/2584))
    - [scrapely](https://github.com/scrapy/scrapely) (was `scrapy.contrib.ibl`, done in [r2586](http://hg.scrapy.org/scrapy/changeset/2586))
- Removed unused function: `scrapy.utils.request.request_info()` ([r2577](http://hg.scrapy.org/scrapy/changeset/2577))
- Removed googledir project from `examples/googledir`. There’s now a new example project called `dirbot` available on GitHub: [https://github.com/scrapy/dirbot](https://github.com/scrapy/dirbot)
- Removed support for default field values in Scrapy items ([r2616](http://hg.scrapy.org/scrapy/changeset/2616))
- Removed experimental crawlspider v2 ([r2632](http://hg.scrapy.org/scrapy/changeset/2632))
- Removed scheduler middleware to simplify architecture. Duplicates filter is now done in the scheduler itself, using the same dupe filtering class as before (`DUPEFILTER_CLASS` setting) ([r2640](http://hg.scrapy.org/scrapy/changeset/2640))
- Removed support for passing urls to `scrapy crawl` command (use `scrapy parse` instead) ([r2704](http://hg.scrapy.org/scrapy/changeset/2704))
- Removed deprecated Execution Queue ([r2704](http://hg.scrapy.org/scrapy/changeset/2704))
- Removed (undocumented) spider context extension (from scrapy.contrib.spidercontext) ([r2780](http://hg.scrapy.org/scrapy/changeset/2780))
- removed `CONCURRENT_SPIDERS` setting (use scrapyd maxproc instead) ([r2789](http://hg.scrapy.org/scrapy/changeset/2789))
- Renamed attributes of core components: downloader.sites -> downloader.slots, scraper.sites -> scraper.slots ([r2717](http://hg.scrapy.org/scrapy/changeset/2717), [r2718](http://hg.scrapy.org/scrapy/changeset/2718))
- Renamed setting `CLOSESPIDER_ITEMPASSED` to [`CLOSESPIDER_ITEMCOUNT`](../topics/extensions.md) ([r2655](http://hg.scrapy.org/scrapy/changeset/2655)). Backward compatibility kept.

## Scrapy 0.12 (2011-01-02)

The numbers like #NNN reference tickets in the old issue tracker (Trac) which is no longer available.

### New features and improvements

- Passed item is now sent in the `item` argument of the [`item_passed`](../topics/signals.md) (#273)
- Added verbose option to `scrapy version` command, useful for bug reports (#298)
- HTTP cache now stored by default in the project data dir (#279)
- Added project data storage directory (#276, #277)
- Documented file structure of Scrapy projects (see command-line tool doc)
- New lxml backend for XPath selectors (#147)
- Per-spider settings (#245)
- Support exit codes to signal errors in Scrapy commands (#248)
- Added `-c` argument to `scrapy shell` command
- Made `libxml2` optional (#260)
- New `deploy` command (#261)
- Added [`CLOSESPIDER_PAGECOUNT`](../topics/extensions.md) setting (#253)
- Added [`CLOSESPIDER_ERRORCOUNT`](../topics/extensions.md) setting (#254)

### Scrapyd changes

- Scrapyd now uses one process per spider
- It stores one log file per spider run, and rotate them keeping the latest 5 logs per spider (by default)
- A minimal web ui was added, available at [http://localhost:6800](http://localhost:6800) by default
- There is now a `scrapy server` command to start a Scrapyd server of the current project

### Changes to settings

- added `HTTPCACHE_ENABLED` setting (False by default) to enable HTTP cache middleware
- changed `HTTPCACHE_EXPIRATION_SECS` semantics: now zero means “never expire”.

### Deprecated/obsoleted functionality

- Deprecated `runserver` command in favor of `server` command which starts a Scrapyd server. See also: Scrapyd changes
- Deprecated `queue` command in favor of using Scrapyd `schedule.json` API. See also: Scrapyd changes
- Removed the !LxmlItemLoader (experimental contrib which never graduated to main contrib)

## Scrapy 0.10 (2010-09-11)

The numbers like #NNN reference tickets in the old issue tracker (Trac) which is no longer available.

### New features and improvements

- New Scrapy service called `scrapyd` for deploying Scrapy crawlers in production (#218) (documentation available)
- Simplified Images pipeline usage which doesn’t require subclassing your own images pipeline now (#217)
- Scrapy shell now shows the Scrapy log by default (#206)
- Refactored execution queue in a common base code and pluggable backends called “spider queues” (#220)
- New persistent spider queue (based on SQLite) (#198), available by default, which allows to start Scrapy in server mode and then schedule spiders to run.
- Added documentation for Scrapy command-line tool and all its available sub-commands. (documentation available)
- Feed exporters with pluggable backends (#197) (documentation available)
- Deferred signals (#193)
- Added two new methods to item pipeline open_spider(), close_spider() with deferred support (#195)
- Support for overriding default request headers per spider (#181)
- Replaced default Spider Manager with one with similar functionality but not depending on Twisted Plugins (#186)
- Split Debian package into two packages - the library and the service (#187)
- Scrapy log refactoring (#188)
- New extension for keeping persistent spider contexts among different runs (#203)
- Added `dont_redirect` request.meta key for avoiding redirects (#233)
- Added `dont_retry` request.meta key for avoiding retries (#234)

### Command-line tool changes

- New `scrapy` command which replaces the old `scrapy-ctl.py` (#199)
  - there is only one global `scrapy` command now, instead of one `scrapy-ctl.py` per project
  - Added `scrapy.bat` script for running more conveniently from Windows
- Added bash completion to command-line tool (#210)
- Renamed command `start` to `runserver` (#209)

### API changes

- `url` and `body` attributes of Request objects are now read-only (#230)
- `Request.copy()` and `Request.replace()` now also copies their `callback` and `errback` attributes (#231)
- Removed `UrlFilterMiddleware` from `scrapy.contrib` (already disabled by default)
- Offsite middleware doesn’t filter out any request coming from a spider that doesn’t have a allowed_domains attribute (#225)
- Removed Spider Manager `load()` method. Now spiders are loaded in the `__init__` method itself.
- Changes to Scrapy Manager (now called “Crawler”):
  : - `scrapy.core.manager.ScrapyManager` class renamed to `scrapy.crawler.Crawler`
    - `scrapy.core.manager.scrapymanager` singleton moved to `scrapy.project.crawler`
- Moved module: `scrapy.contrib.spidermanager` to `scrapy.spidermanager`
- Spider Manager singleton moved from `scrapy.spider.spiders` to the `spiders` attribute of `scrapy.project.crawler` singleton.
- moved Stats Collector classes: (#204)
  : - `scrapy.stats.collector.StatsCollector` to `scrapy.statscol.StatsCollector`
    - `scrapy.stats.collector.SimpledbStatsCollector` to `scrapy.contrib.statscol.SimpledbStatsCollector`
- default per-command settings are now specified in the `default_settings` attribute of command object class (#201)
- changed arguments of Item pipeline `process_item()` method from `(spider, item)` to `(item, spider)`
  : - backward compatibility kept (with deprecation warning)
- moved `scrapy.core.signals` module to `scrapy.signals`
  : - backward compatibility kept (with deprecation warning)
- moved `scrapy.core.exceptions` module to `scrapy.exceptions`
  : - backward compatibility kept (with deprecation warning)
- added `handles_request()` class method to `BaseSpider`
- dropped `scrapy.log.exc()` function (use `scrapy.log.err()` instead)
- dropped `component` argument of `scrapy.log.msg()` function
- dropped `scrapy.log.log_level` attribute
- Added `from_settings()` class methods to Spider Manager, and Item Pipeline Manager

### Changes to settings

- Added `HTTPCACHE_IGNORE_SCHEMES` setting to ignore certain schemes on !HttpCacheMiddleware (#225)
- Added `SPIDER_QUEUE_CLASS` setting which defines the spider queue to use (#220)
- Added `KEEP_ALIVE` setting (#220)
- Removed `SERVICE_QUEUE` setting (#220)
- Removed `COMMANDS_SETTINGS_MODULE` setting (#201)
- Renamed `REQUEST_HANDLERS` to `DOWNLOAD_HANDLERS` and make download handlers classes (instead of functions)

## Scrapy 0.9 (2010-06-28)

The numbers like #NNN reference tickets in the old issue tracker (Trac) which is no longer available.

### New features and improvements

- Added SMTP-AUTH support to scrapy.mail
- New settings added: `MAIL_USER`, `MAIL_PASS` ([r2065](http://hg.scrapy.org/scrapy/changeset/2065) | #149)
- Added new scrapy-ctl view command - To view URL in the browser, as seen by Scrapy ([r2039](http://hg.scrapy.org/scrapy/changeset/2039))
- Added web service for controlling Scrapy process (this also deprecates the web console. ([r2053](http://hg.scrapy.org/scrapy/changeset/2053) | #167)
- Support for running Scrapy as a service, for production systems ([r1988](http://hg.scrapy.org/scrapy/changeset/1988), [r2054](http://hg.scrapy.org/scrapy/changeset/2054), [r2055](http://hg.scrapy.org/scrapy/changeset/2055), [r2056](http://hg.scrapy.org/scrapy/changeset/2056), [r2057](http://hg.scrapy.org/scrapy/changeset/2057) | #168)
- Added wrapper induction library (documentation only available in source code for now). ([r2011](http://hg.scrapy.org/scrapy/changeset/2011))
- Simplified and improved response encoding support ([r1961](http://hg.scrapy.org/scrapy/changeset/1961), [r1969](http://hg.scrapy.org/scrapy/changeset/1969))
- Added `LOG_ENCODING` setting ([r1956](http://hg.scrapy.org/scrapy/changeset/1956), documentation available)
- Added `RANDOMIZE_DOWNLOAD_DELAY` setting (enabled by default) ([r1923](http://hg.scrapy.org/scrapy/changeset/1923), doc available)
- `MailSender` is no longer IO-blocking ([r1955](http://hg.scrapy.org/scrapy/changeset/1955) | #146)
- Linkextractors and new Crawlspider now handle relative base tag urls ([r1960](http://hg.scrapy.org/scrapy/changeset/1960) | #148)
- Several improvements to Item Loaders and processors ([r2022](http://hg.scrapy.org/scrapy/changeset/2022), [r2023](http://hg.scrapy.org/scrapy/changeset/2023), [r2024](http://hg.scrapy.org/scrapy/changeset/2024), [r2025](http://hg.scrapy.org/scrapy/changeset/2025), [r2026](http://hg.scrapy.org/scrapy/changeset/2026), [r2027](http://hg.scrapy.org/scrapy/changeset/2027), [r2028](http://hg.scrapy.org/scrapy/changeset/2028), [r2029](http://hg.scrapy.org/scrapy/changeset/2029), [r2030](http://hg.scrapy.org/scrapy/changeset/2030))
- Added support for adding variables to telnet console ([r2047](http://hg.scrapy.org/scrapy/changeset/2047) | #165)
- Support for requests without callbacks ([r2050](http://hg.scrapy.org/scrapy/changeset/2050) | #166)

### API changes

- Change `Spider.domain_name` to `Spider.name` (SEP-012, [r1975](http://hg.scrapy.org/scrapy/changeset/1975))
- `Response.encoding` is now the detected encoding ([r1961](http://hg.scrapy.org/scrapy/changeset/1961))
- `HttpErrorMiddleware` now returns None or raises an exception ([r2006](http://hg.scrapy.org/scrapy/changeset/2006) | #157)
- `scrapy.command` modules relocation ([r2035](http://hg.scrapy.org/scrapy/changeset/2035), [r2036](http://hg.scrapy.org/scrapy/changeset/2036), [r2037](http://hg.scrapy.org/scrapy/changeset/2037))
- Added `ExecutionQueue` for feeding spiders to scrape ([r2034](http://hg.scrapy.org/scrapy/changeset/2034))
- Removed `ExecutionEngine` singleton ([r2039](http://hg.scrapy.org/scrapy/changeset/2039))
- Ported `S3ImagesStore` (images pipeline) to use boto and threads ([r2033](http://hg.scrapy.org/scrapy/changeset/2033))
- Moved module: `scrapy.management.telnet` to `scrapy.telnet` ([r2047](http://hg.scrapy.org/scrapy/changeset/2047))

### Changes to default settings

- Changed default `SCHEDULER_ORDER` to `DFO` ([r1939](http://hg.scrapy.org/scrapy/changeset/1939))

## Scrapy 0.8 (2009-12-12)

The numbers like #NNN reference tickets in the old issue tracker (Trac) which is no longer available.

### New features

- Added DEFAULT_RESPONSE_ENCODING setting ([r1809](http://hg.scrapy.org/scrapy/changeset/1809))
- Added `dont_click` argument to `FormRequest.from_response()` method ([r1813](http://hg.scrapy.org/scrapy/changeset/1813), [r1816](http://hg.scrapy.org/scrapy/changeset/1816))
- Added `clickdata` argument to `FormRequest.from_response()` method ([r1802](http://hg.scrapy.org/scrapy/changeset/1802), [r1803](http://hg.scrapy.org/scrapy/changeset/1803))
- Added support for HTTP proxies (`HttpProxyMiddleware`) ([r1781](http://hg.scrapy.org/scrapy/changeset/1781), [r1785](http://hg.scrapy.org/scrapy/changeset/1785))
- Offsite spider middleware now logs messages when filtering out requests ([r1841](http://hg.scrapy.org/scrapy/changeset/1841))

### Backward-incompatible changes

- Changed `scrapy.utils.response.get_meta_refresh()` signature ([r1804](http://hg.scrapy.org/scrapy/changeset/1804))
- Removed deprecated `scrapy.item.ScrapedItem` class - use `scrapy.item.Item instead` ([r1838](http://hg.scrapy.org/scrapy/changeset/1838))
- Removed deprecated `scrapy.xpath` module - use `scrapy.selector` instead. ([r1836](http://hg.scrapy.org/scrapy/changeset/1836))
- Removed deprecated `core.signals.domain_open` signal - use `core.signals.domain_opened` instead ([r1822](http://hg.scrapy.org/scrapy/changeset/1822))
- `log.msg()` now receives a `spider` argument ([r1822](http://hg.scrapy.org/scrapy/changeset/1822))
  : - Old domain argument has been deprecated and will be removed in 0.9. For spiders, you should always use the `spider` argument and pass spider references. If you really want to pass a string, use the `component` argument instead.
- Changed core signals `domain_opened`, `domain_closed`, `domain_idle`
- Changed Item pipeline to use spiders instead of domains
  : - The `domain` argument of  `process_item()` item pipeline method was changed to  `spider`, the new signature is: `process_item(spider, item)` ([r1827](http://hg.scrapy.org/scrapy/changeset/1827) | #105)
    - To quickly port your code (to work with Scrapy 0.8) just use `spider.domain_name` where you previously used `domain`.
- Changed Stats API to use spiders instead of domains ([r1849](http://hg.scrapy.org/scrapy/changeset/1849) | #113)
  : - `StatsCollector` was changed to receive spider references (instead of domains) in its methods (`set_value`, `inc_value`, etc).
    - added `StatsCollector.iter_spider_stats()` method
    - removed `StatsCollector.list_domains()` method
    - Also, Stats signals were renamed and now pass around spider references (instead of domains). Here’s a summary of the changes:
    - To quickly port your code (to work with Scrapy 0.8) just use `spider.domain_name` where you previously used `domain`. `spider_stats` contains exactly the same data as `domain_stats`.
- `CloseDomain` extension moved to `scrapy.contrib.closespider.CloseSpider` ([r1833](http://hg.scrapy.org/scrapy/changeset/1833))
  : - Its settings were also renamed:
      : - `CLOSEDOMAIN_TIMEOUT` to `CLOSESPIDER_TIMEOUT`
        - `CLOSEDOMAIN_ITEMCOUNT` to `CLOSESPIDER_ITEMCOUNT`
- Removed deprecated `SCRAPYSETTINGS_MODULE` environment variable - use `SCRAPY_SETTINGS_MODULE` instead ([r1840](http://hg.scrapy.org/scrapy/changeset/1840))
- Renamed setting: `REQUESTS_PER_DOMAIN` to `CONCURRENT_REQUESTS_PER_SPIDER` ([r1830](http://hg.scrapy.org/scrapy/changeset/1830), [r1844](http://hg.scrapy.org/scrapy/changeset/1844))
- Renamed setting: `CONCURRENT_DOMAINS` to `CONCURRENT_SPIDERS` ([r1830](http://hg.scrapy.org/scrapy/changeset/1830))
- Refactored HTTP Cache middleware
- HTTP Cache middleware has been heavily refactored, retaining the same functionality except for the domain sectorization which was removed. ([r1843](http://hg.scrapy.org/scrapy/changeset/1843) )
- Renamed exception: `DontCloseDomain` to `DontCloseSpider` ([r1859](http://hg.scrapy.org/scrapy/changeset/1859) | #120)
- Renamed extension: `DelayedCloseDomain` to `SpiderCloseDelay` ([r1861](http://hg.scrapy.org/scrapy/changeset/1861) | #121)
- Removed obsolete `scrapy.utils.markup.remove_escape_chars` function - use `scrapy.utils.markup.replace_escape_chars` instead ([r1865](http://hg.scrapy.org/scrapy/changeset/1865))

## Scrapy 0.7 (2009-10-07)

First release of Scrapy.