How To Scrape An Interactive Charts With Scrapy?

I'm using scrapy to scrape and crawl webpages. I am interested in how to scrape this page. As you can see there are several charts. But when I look at the source code, I do not fin

Solution 1:

Data is on your webpage. Check in script tag variables var cote_data_1, var cote_data_2, etc. They should be available without JS.

Solution 2:

Here is the code that scrapes the first chart data:

import scrapy
import ast


deffind_between(s, start, end):
  return (s.split(start))[1].split(end)[0]


classCanalTurfSpider(scrapy.Spider):
    name = "CanalTurfSpider"
    start_urls = ['https://www.canalturf.com/cotes/2019-04-15/''maisons-laffitte/185850_prix-des-ecuries-du-chateau.html']
    
    defparse(self, response):
        data = response.xpath('//script').extract()[-1]
        chart1_data = find_between(data, "var cote_data_1 = ", ";")
        chart1_data = ast.literal_eval(chart1_data)
        yield {
            "chart1_data": chart1_data
        }

Output:

{'chart1_data': [{'elapsed': '12:25', 'value': 9.3}, {'elapsed': '12:35', 'value': 9.7}, {'elapsed': '12:45', 'value': 10}, {'elapsed': '12:55', 'value': 10.1}, {'elapsed': '13:05', 'v
alue': 10.6}, {'elapsed': '13:15', 'value': 10.6}, {'elapsed': '13:25', 'value': 11.2}, {'elapsed': '13:35', 'value': 11.3}, {'elapsed': '13:45', 'value': 13.1}, {'elapsed': '13:55', '
value': 14.7}, {'elapsed': '14:05', 'value': 18.8}, {'elapsed': '14:15', 'value': 18.8}]}

you can find variables in last script block.