The problem is that I can't debug Scrapy crawlers in VS Code. The problem is that always when I start debugging it breaks on one of my imports. Of course, I played a lot with that import in order to…
How do I extract the text from the last <li> in the following snippet? (Černošice.) <footer class="SearchResultCard__footer"> <ul class="SearchResultCard__footerList"> <li class="SearchResultCard__footerItem"> <svg xmlns="http://www.w3.org/2000/svg" width="16" height="16" viewBox="0 0 24 24" fill="none" id="7c37b661a1f4030a0673d3e5cb419678" aria-hidden="true"> <path fill-rule="evenodd" clip-rule="evenodd" d="M6.16146 2H9.83854C10.3657 1.99998 10.8205 1.99997…
Hi I'm trying to web scrape (with Scrapy) this website https://www.vaniercollege.qc.ca/sports-recreation/weekly-schedule/ from this script below script.py import scrapy from scrapy.crawler import CrawlerProcess from threading import Thread class CourtSpider(scrapy.Spider): name = 'full_page' allowed_domains = ['vaniercollege.qc.ca'] start_urls = ['https://www.vaniercollege.qc.ca/sports-recreation/weekly-schedule/'] def parse(self, response):…
I want to extract info from this website using Scrapy. But the info I need is in a JSON file; and this JSON file has unwanted literal newlines characters in only the description section. Here is an example page and…
I'm experiencing difficulties in accessing a ScrapyRT service running on specific ports within a Kubernetes pod. My setup includes a Kubernetes cluster with a pod running a Scrapy application, which uses ScrapyRT to listen for incoming requests on designated ports.…
please check this link https://maroof.sa/businesses. it is a link for website from which i want to extract links. for example if you scroll down you would find a name for store "Marwa store" if you click on this card this…
I wrote the below python script to return the item name, price, and link for items listed on https://shop.doverstreetmarket.com/collections/shops-noah import scrapy class DSMUKSpider(scrapy.Spider): name = 'dsmuk' start_urls = ['https://shop.doverstreetmarket.com/collections/shops-noah'] def parse(self, response): for dsmuk_product in response.css('article.h-full'): try: yield { 'name':…
I am trying to scrape a page that contains specific info. The url:https://www.artisans-du-batiment.com/trouver-un-artisan-qualifie/?job=Charpentier&place=35000%2F35900 I want to select a class for each carpenter, so I try response.css('div.a-artisanTease to-animate'), but it gives no selection. What might be the problem? Thanks. I've tried…
We have a vendor that sells many products that we want to include. they unfortunately do not have an API setup to retrieve information. they do however give us product lists that show just the SKU of an item. I…
Here's a simple scrapy spider that anyone can use for testing. from scrapy.utils.response import open_in_browser import scrapy import json class TestSpider(scrapy.Spider): name = "test-spider" allowed_domains = ["shopee.ph"] shopee_cookies = '[{"name": "csrftoken", "value": "RvxBdTixvBfdTR3xfQwbcYippqz8jEbF", "domain": "shopee.ph", "path": "/", "expires": -1, "httpOnly":…