之前爬链家二手房的时候说过，他的下一页链接是通过js来动态加载的，所以如果我们使用普通方法的话，是没办法爬取的，然后我们就可以尝试使用splash。

首先安装docker然后安装splash，之后打开链接http://localhost:8050/然后输入网址可以查看源代码，发现可以获取到页面链接，那么就可以通过splash智能翻页了。

首先我们要更改一下setting文件

SPLASH_URL = "http://localhost:8050/"
DUPEFILTER_CLASS = "scrapy_splash.SplashAwareDupeFilter"
HTTPCACHE_STORAGE = "scrapy_splash.SplashAwareFSCacheStorage"

SPIDER_MIDDLEWARES = {
    'scrapy_splash.SplashDeduplicateArgsMiddleware': 100,
   # 'lianjia.middlewares.LianjiaSpiderMiddleware': 543,
}

DOWNLOADER_MIDDLEWARES = {
    'scrapy_splash.SplashCookiesMiddleware': 723,
    'scrapy_splash.SplashMiddleware': 728,
    'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 823,
   # 'lianjia.middlewares.LianjiaDownloaderMiddleware': 543,
}

from scrapy_splash import SplashRequest
from scrapy.spiders import Spider
from scrapy import Request
from lianjia.items import LianjiaItem

lua_script = """
    function main(splash, args)
      assert(splash:go(args.url))
      assert(splash:wait(0.5))
      return {
        html = splash:html()
      }
end
"""

class LianjiaSpider(Spider):
    name = "lianjia"
    url = 'https://yt.lianjia.com/ershoufang/'
    def start_requests(self):
        yield SplashRequest(self.url,
                            callback=self.parse,
                            endpoint='execute',
                            args={
                                'lua_source': lua_script,
                                'images': 0,
                                'wait': 3
                            },
                            cache_args=['lua_source'])
    def parse(self, response):
        item = LianjiaItem()
        list_selector = response.xpath("//ul[@class='sellListContent']/li")
        for one_selector in list_selector:
            try:
                name = one_selector.xpath("div[@class='info clear']/div[@class='title']/a/text()").extract()[0]
                price = one_selector.xpath("div[@class='info clear']/div[@class='priceInfo']/div[1]/span/text()").extract()[0]
                item['name'] = name
                item['price'] = price

                yield item
            except:
                pass
        next_url = response.xpath("//div[@class='page-box house-lst-page-box']/a[last()]/text()").extract()[0]
        if next_url == "下一页":
            next_url = response.xpath("//div[@class='page-box house-lst-page-box']/a[last()]/@href").extract()[0]
            next_url = "https://yt.lianjia.com"+next_url
            yield SplashRequest(next_url,
                                callback=self.parse,
                                endpoint='execute',
                                args={
                                    'lua_source': lua_script,
                                    'images': 0,
                                    'wait': 3
                                },
                                cache_args=['lua_source'])

splash就要通过splashrequest来进行请求。
然后获取页面信息进行翻页获取信息。