How To Avoid Redirection Of The Webcrawler To The Mobile Edition?

I subclassed a CrawlSpider and want to extract data from website. However, I always get redirected to the site's mobile version. I tried to change the USER_AGENT variable in scrapy

Solution 1:

There are two types of redirection supported in Scrapy:

• RedirectMiddleware - Handle redirection of requests based on response status
• MetaRefreshMiddleware - Handle redirection of requests based on meta-refresh html tag

So, maybe your html page uses second type of redirection?

See also:

• http://doc.scrapy.org/en/latest/topics/downloader-middleware.html#redirectmiddleware-settings
• http://doc.scrapy.org/en/latest/topics/downloader-middleware.html#metarefreshmiddleware-settings