Ebay API – Python BeautifulSoup find all error (object has no attribute)

Most likely you get a CAPTCHA or IP rate limit. Ways to avoid being blocked.

If you need to extract all results from all pages using pagination, the solution to this would be to use an non-token pagination and test for something (button, element) that will result in an exit:

if soup.select_one(".pagination__next"):   # checking for 'next page' button
    params['_pgn'] += 1                    # if there is a button, it will go to the next page
else:                                      # otherwise, the loop exits
    break

You can also add a condition for exiting the loop by the number of retrieved pages by adding a limit:

limit = 5                     # page limit

# other code

if params['_pgn'] == limit:   # if the page number is equal to the specified limit, the loop is terminated
    break

Code example with pagination in the online IDE.

from bs4 import BeautifulSoup
import requests, json, lxml

# https://requests.readthedocs.io/en/latest/user/quickstart/#custom-headers
headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36",
}
   
params = {
    "_nkw": "iphone_14",    # search query example
    "LH_Sold": "1",         # shows sold items
    "_pgn": 1               # page number
}

data = []
limit = 5                 # page limit (if needed)
while True:
    page = requests.get("https://www.ebay.co.uk/sch/i.html", params=params, headers=headers, timeout=30)
    soup = BeautifulSoup(page.text, "lxml")
    
    print(f"Extracting page: {params['_pgn']}")

    print("-" * 10)
    
    for products in soup.select(".s-item__info"):
        title = products.select_one(".s-item__title span").text
        price = products.select_one(".s-item__price").text
        
        data.append({
          "title" : title,
          "price" : price
        })

    if params['_pgn'] == limit:
       break
    if soup.select_one(".pagination__next"):
        params['_pgn'] += 1
    else:
        break

print(json.dumps(data, indent=2, ensure_ascii=False))

Example output:

[
  {
    "title": "Case For iPhone 11 Pro Max 14Pro 8 7  SE 2022  Shockproof Silicone Cover colours",
    "price": "£3.99"
  },
  {
    "title": "Ring Holder Magnetic Shockproof Case Cover For iPhone  14Pro Max 11 XR  XS 12 13",
    "price": "£5.99 to £6.99"
  },
  {
    "title": "Apple iPhone 14 - 128GB - Space Black (Unlocked) A2890 (GSM)",
    "price": "£641.95"
  },
  other results ...
]

As an alternative, you can use Ebay Organic Results API from SerpApi. It’s a paid API with a free plan that handles blocks and parsing on their backend.

Example code with pagination:

from serpapi import EbaySearch
import json

params = {
    "api_key": "...",                 # serpapi key, https://serpapi.com/manage-api-key   
    "engine": "ebay",                 # search engine
    "ebay_domain": "ebay.co.uk",      # ebay domain
    "_nkw": "iphone_14",              # search query
    "LH_Sold": "1",                   # shows sold items
    "_pgn": 1                         # page number
}

search = EbaySearch(params)           # where data extraction happens

limit = 5
page_num = 0
data = []

while True:
    results = search.get_dict()     # JSON -> Python dict

    if "error" in results:
        print(results["error"])
        break
    
    for organic_result in results.get("organic_results", []):
        title = organic_result.get("title")
        price = organic_result.get("price")

        data.append({
          "title" : title,
          "price" : price
        })
                    
    page_num += 1
    print(page_num)

    if params['_pgn'] == limit:
       break
    if "next" in results.get("pagination", {}):
        params['_pgn'] += 1
    else:
        break

print(json.dumps(data, indent=2, ensure_ascii=False))

Output:

[
  {
    "title": "Apple iPhone 14 Plus Midnight - 512GB - Unlocked - MINT CONDITION",
    "price": {
      "raw": "£749.99",
      "extracted": 749.99
    }
  },
  {
    "title": "New listingApple iPhone 14 Plus (PRODUCT)RED - 128GB (Unlocked)",
    "price": {
      "raw": "£750.00",
      "extracted": 750.0
    }
  other results ...
]

There’s a 13 ways to scrape any public data from any website blog post if you want to know more about website scraping.