Hello I am trying to get Flickr public images from flicker group and I am able to parse the html and get the image href however struggling to find a way to get all the images from a page as we scroll the page down. My below code only returns all the hrefs for images on front page but how can I get all the href after scrolling to the bottom.
from bs4 import BeautifulSoup
import urllib.request
from selenium import webdriver
import time
op = webdriver.ChromeOptions()
op.add_argument('headless')
driver = webdriver.Chrome(options=op)
url = "https://www.flickr.com/groups/allfreepictures/pool/page3041"
driver.get(url=url)
html1 = driver.page_source
soup = BeautifulSoup(html1, 'html.parser')
image_urls = [link['href'] for link in soup.findAll("a", {"class": "overlay"})]
print(image_urls)
2
Answers
By sending page down key, maybe you can get what you want. Try this:
You should first try scrolling to the bottom of the page to make sure to load all the available images on the page.
Here’s how you try:
output: