Can scrapy be used to scrape dynamic content from websites that are using AJAX?

Can scrapy be used to scrape dynamic content from websites that are using AJAX?

I have recently been learning Python and am dipping my hand into building a web-scraper. It's nothing fancy at all; its only purpose is to get the data off of a betting website and have this data put into Excel.

Most of the issues are solvable and I'm having a good little mess around. However I'm hitting a massive hurdle over one issue. If a site loads a table of horses and lists current betting prices this information is not in any source file. The clue is that this data is live sometimes, with the numbers being updated obviously from some remote server. The HTML on my PC simply has a hole where their servers are pushing through all the interesting data that I need.

Now my experience with dynamic web content is low, so this thing is something I'm having trouble getting my head around.

I think Java or Javascript is a key, this pops up often.

The scraper is simply a odds comparison engine. Some sites have APIs but I need this for those that don't. I'm using the scrapy library with Python 2.7

I do apologize if this question is too open-ended. In short, my question is: how can scrapy be used to scrape this dynamic data so that I can use it? So that I can scrape this betting odds data in real-time?

Cheers people :)

Answer by ski for Can scrapy be used to scrape dynamic content from websites that are using AJAX?

Webkit based browsers (like Google Chrome or Safari) has built-in developer tools. In Chrome you can open it Menu->Tools->Developer Tools. The Network tab allows you to see all information about every request and response:

In the bottom of the picture you can see that I've filtered request down to XHR - these are requests made by javascript code.

Tip: log is cleared every time you load a page, at the bottom of the picture, the black dot button will preserve log.

After analyzing requests and responses you can simulate these requests from your web-crawler and extract valuable data. In many cases it will be easier to get your data than parsing HTML, because that data does not contain presentation logic and is formatted to be accessed by javascript code.

Firefox has similar extension, it is called firebug. Some will argue that firebug is even more powerful but I like the simplicity of webkit.

Answer by Badarau Petru for Can scrapy be used to scrape dynamic content from websites that are using AJAX?

Here is a simple example of using scrapy with ajax request. Let see the site http://www.rubin-kazan.ru/guestbook.html All messages are loaded with an ajax request. My goal is to fetch this messages with all their attributes (author, date, ...).

When I analyse the source code of the page I can't see all these messages because the web page use ajax technology. But I can with Firebug from Mozila Firefox (or an analogy instrument in other browser) to analyse the Http request that generate the messages on the web page.

For this purpose I don't reload all page but only the part of page that contain messages. For this purpose I click an arbitrary number of page on the bottom and I observe the HTTP request that is responsible about message body

After finish I analyse the headers of request (I must quote that this url I'll extract from source page from var section, see the code below).

and the form data content of request (the Http method is "Post")

and the content of response, which is an Json file,

which present all information I'm looking for.

From now I must implement all this knowledge in scrapy. Let's define the spider for this purpose.

  class spider(BaseSpider):        name = 'RubiGuesst'        start_urls = ['http://www.rubin-kazan.ru/guestbook.html']        def parse(self, response):        url_list_gb_messages = re.search(r'url_list_gb_messages="(.*)"', response.body).group(1)        yield FormRequest('http://www.rubin-kazan.ru' + url_list_gb_messages, callback=self.RubiGuessItem, formdata={'page': str(page + 1), 'uid': ''})      def RubiGuessItem(self, response):         json_file = response.body  

In parse function I have the response for first request. In RubiGuessItem I have the json file with all information.

I hope this response will help you. Best regards.

Answer by A T for Can scrapy be used to scrape dynamic content from websites that are using AJAX?

Many times when crawling we run into problems where content that is rendered on the page is generated with Javascript and therefore scrapy is unable to crawl for it (eg. ajax requests, jQuery craziness).

However, if you use Scrapy along with the web testing framework Selenium then we are able to crawl anything displayed in a normal web browser.

Some things to note:

• You must have the Python version of Selenium RC installed for this to work, and you must have set up Selenium properly. Also this is just a template crawler. You could get much crazier and more advanced with things but I just wanted to show the basic idea. As the code stands now you will be doing two requests for any given url. One request is made by Scrapy and the other is made by Selenium. I am sure there are ways around this so that you could possibly just make Selenium do the one and only request but I did not bother to implement that and by doing two requests you get to crawl the page with Scrapy too.

• This is quite powerful because now you have the entire rendered DOM available for you to crawl and you can still use all the nice crawling features in Scrapy. This will make for slower crawling of course but depending on how much you need the rendered DOM it might be worth the wait.

  from scrapy.contrib.spiders import CrawlSpider, Rule  from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor  from scrapy.selector import HtmlXPathSelector  from scrapy.http import Request    from selenium import selenium    class SeleniumSpider(CrawlSpider):      name = "SeleniumSpider"      start_urls = ["http://www.domain.com"]        rules = (          Rule(SgmlLinkExtractor(allow=('\.html', )), callback='parse_page',follow=True),      )        def __init__(self):          CrawlSpider.__init__(self)          self.verificationErrors = []          self.selenium = selenium("localhost", 4444, "*chrome", "http://www.domain.com")          self.selenium.start()        def __del__(self):          self.selenium.stop()          print self.verificationErrors          CrawlSpider.__del__(self)        def parse_page(self, response):          item = Item()            hxs = HtmlXPathSelector(response)          #Do some XPath selection with Scrapy          hxs.select('//div').extract()            sel = self.selenium          sel.open(response.url)            #Wait for javscript to load in Selenium          time.sleep(2.5)            #Do some crawling of javascript created content with Selenium          sel.get_text("//div")          yield item    # Snippet imported from snippets.scrapy.org (which no longer works)  # author: wynbennett  # date  : Jun 21, 2011  

Reference: http://snipplr.com/view/66998/

Answer by rocktheartsm4l for Can scrapy be used to scrape dynamic content from websites that are using AJAX?

Another solution would be to implement a download handler or download handler middleware. The following is an example of middleware using selenium with headless phantomjs webdriver:

class JsDownload(object):    @check_spider_middleware  def process_request(self, request, spider):      driver = webdriver.PhantomJS(executable_path='D:\phantomjs.exe')      driver.get(request.url)      return HtmlResponse(request.url, encoding='utf-8', body=driver.page_source.encode('utf-8'))  

I wanted to ability to tell different spiders which middleware to use so I implemented this wrapper:

def check_spider_middleware(method):  @functools.wraps(method)  def wrapper(self, request, spider):      msg = '%%s %s middleware step' % (self.__class__.__name__,)      if self.__class__ in spider.middleware:          spider.log(msg % 'executing', level=log.DEBUG)          return method(self, request, spider)      else:          spider.log(msg % 'skipping', level=log.DEBUG)          return None    return wrapper  

settings.py:

DOWNLOADER_MIDDLEWARES = {'MyProj.middleware.MiddleWareModule.MiddleWareClass': 500}  

for wrapper to work all spiders must have at minimum:

middleware = set([])  

to include a middleware:

middleware = set([MyProj.middleware.ModuleName.ClassName])  

The main advantage to implementing it this way rather than in the spider is that you only end up making one request. In A T's solution for example: The download handler processes the request and then hands off the response to the spider. The spider then makes a brand new request in it's parse_page function -- That's two requests for the same content.

Cheers!

Answer by narko for Can scrapy be used to scrape dynamic content from websites that are using AJAX?

I handle the ajax request by using Selenium and the Firefox web driver. It is not that fast if you need the crawler as a daemon, but much better than any manual solution. I wrote a short tutorial here for reference: http://www.6020peaks.com/2014/12/how-to-scrape-hidden-web-data-with-python/

Answer by Ivan Chaer for Can scrapy be used to scrape dynamic content from websites that are using AJAX?

I was using a custom downloader middleware, but wasn't very happy with it, as I didn't manage to make the cache work with it.

A better approach was to implement a custom download handler.

There is a working example here. It looks like this:

# encoding: utf-8  from __future__ import unicode_literals    from scrapy import signals  from scrapy.signalmanager import SignalManager  from scrapy.responsetypes import responsetypes  from scrapy.xlib.pydispatch import dispatcher  from selenium import webdriver  from six.moves import queue  from twisted.internet import defer, threads  from twisted.python.failure import Failure      class PhantomJSDownloadHandler(object):        def __init__(self, settings):          self.options = settings.get('PHANTOMJS_OPTIONS', {})            max_run = settings.get('PHANTOMJS_MAXRUN', 10)          self.sem = defer.DeferredSemaphore(max_run)          self.queue = queue.LifoQueue(max_run)            SignalManager(dispatcher.Any).connect(self._close, signal=signals.spider_closed)        def download_request(self, request, spider):          """use semaphore to guard a phantomjs pool"""          return self.sem.run(self._wait_request, request, spider)        def _wait_request(self, request, spider):          try:              driver = self.queue.get_nowait()          except queue.Empty:              driver = webdriver.PhantomJS(**self.options)            driver.get(request.url)          # ghostdriver won't response when switch window until page is loaded          dfd = threads.deferToThread(lambda: driver.switch_to.window(driver.current_window_handle))          dfd.addCallback(self._response, driver, spider)          return dfd        def _response(self, _, driver, spider):          body = driver.execute_script("return document.documentElement.innerHTML")          if body.startswith(""):  # cannot access response header in Selenium              body = driver.execute_script("return document.documentElement.textContent")          url = driver.current_url          respcls = responsetypes.from_args(url=url, body=body[:100].encode('utf8'))          resp = respcls(url=url, body=body, encoding="utf-8")            response_failed = getattr(spider, "response_failed", None)          if response_failed and callable(response_failed) and response_failed(resp, driver):              driver.close()              return defer.fail(Failure())          else:              self.queue.put(driver)              return defer.succeed(resp)        def _close(self):          while not self.queue.empty():              driver = self.queue.get_nowait()              driver.close()  

Suppose your scraper is called "scraper". If you put the mentioned code inside a file called handlers.py on the root of the "scraper" folder, then you could add to your settings.py:

DOWNLOAD_HANDLERS = {      'http': 'scraper.handlers.PhantomJSDownloadHandler',      'https': 'scraper.handlers.PhantomJSDownloadHandler',  }  

And voil?, the JS parsed DOM, with scrapy cache, retries, etc.

Fatal error: Call to a member function getElementsByTagName() on a non-object in D:\XAMPP INSTALLASTION\xampp\htdocs\endunpratama9i\www-stackoverflow-info-proses.php on line 72