Redis – I have 12000 known URLs, what is the fastest way to scrape them with Python?

I doubt you will find a much faster way than scrapy. It has great tools for crawling a site but it can also just be used for scraping a list of known url:s. And it is useful for scraping jsons too. Just make sure you use concurrent requests for processes several pages at the same time. If you risk getting blocked because of multiple requests in a short period of time you can use rotating proxies kike https://github.com/TeamHG-Memex/scrapy-rotating-proxies or use a scraping VPN like crawlera. 4 hours for only 12k of url:s sound like alot.

How are you scraping the json files with Scrapy?

This code would scrape a json-file from the swedish innovation agency Vinnova containg all projects that been granted financing from the agency and output the titles of all the projects:

import scrapy
import json


class TestscraperSpider(scrapy.Spider):
    name = 'testScraper'
    allowed_domains = ['vinnova.se']
    start_urls = [
        'https://www.vinnova.se/sok-finansiering/hitta-finansiering/search/']

    def parse(self, response):

        jsonresponse = json.loads(response.body_as_unicode())
        titles = [project['Heading']
                  for project in jsonresponse['FindHitList']]
        yield {"titles": titles}

If you have more than one json-file to scrape you can just add more url:s in the list. You can do this in three different ways.

1. Manual adding

You can just copy and paste more url:s into the list. Probably not the best way to do it if you got 12k of url:s.

start_urls = [
        'domain.com/link1', 'domain.com/link2', 'domain.com/link3', 'domain.com/link4',]

2. Get start url:s from an external source

You can just override the start_urls by writing custom init like this:

def __init__(self):
    # Get list of urls from external source 
    self.start_urls = data_external

3. Use a custom method for requesting url:s

Here you just bypass start_url:s and just call scrapy:s request method manually for each link.

from scrapy.http import Request

def start_requests(self):
    # Get url:s from external source. 
    for url in urls:
        yield Request(url)

In your case you can probably use either 2 or 3. Shouldn’t matter when it is that few url:s.