I am working in Python web scrapping using beautiful soup. I couldn’t extract ‘href’ tags and save them to a list.
[<div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/358/40440">React JS Developer</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/391/40439">React.js Developers</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/358/40438">Node JS Developer</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/358/40437">React Native Developer</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/391/40436">Python Full Stack Developers</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/469/40435">SEO SPECIALIST</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/469/40434">VR Designer</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/469/40433">Automation Tester</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/403/40432">Junior Frontend Developer (ReactJS)</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/209/40431">Software Test Engineer</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/249/40430">Senior UI Developer (React )</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/249/40429">UI/UX Senior architect</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/209/40428">Front End Developer</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/249/40427">Senior ADF & Power BI Expert</a></div>, <div
class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/209/40426">Software Engineer/ Senior Software Engineer - Odoo</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/209/40425">Team Lead - Php</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/209/40424">Lead - Automation Testing</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/209/40423">Software Engineer/ Senior Software Engineer - PHP</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/209/40422">Software Engineer/Senior Software Engineer - Moodle</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/209/40421">Senior Software Engineer - React Native</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/209/40420">Team Lead-React</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/159/40419">iOS Developers</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/159/40418">Flutter Developers</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/159/40417">React Native Developers</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/159/40416">Full Stack Developers</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/159/40415">Angular Developers</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/159/40414">React JS Developers</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/159/40413">Node JS Developers</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/159/40412">Junior Business Development Executive</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/209/40411">Team Lead - Node</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/209/40410">Team Lead- Ruby on Rails</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/304/40409">Facility in Charge</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/209/40408">Software Engineer/ Senior Software Engineer - Vue</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/209/40407">Lead- UI/UX Designer</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/209/40406">Software Engineer/ Senior Software Engineer - React
Js</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/209/40405">Software Engineer/ Senior Software Engineer - Node </a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/209/40404">Team Lead - GoLang</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/209/40403">Associate - Business Development.</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/209/40402">Team Lead - React Native</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/209/40401">Lead - Business Analyst</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/209/40400">Software Engineer/Sr.Software Engineer- IOS</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/209/40399">Assistant Manager – Business Development </a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/209/40398">Marketing Executive</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/209/40397">Associate Software Engineer-Freshers</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/209/40396">Software Engineer/Senior Software Engineer-Ruby on Rails</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/209/40395">Senior Software Engineer/ Team Lead - Magento</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/209/40394">Software Engineer/Senior Software Engineer - Java</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/290/40393">SENIOR FULL STACK ENIGNEER - NODEJS/REACTJS</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/290/40392">Front End Developer – ReactJS</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/209/40391">Software Engineer/Senior Software Engineer - .Net</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/262/40390">Administration Executive</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/258/40389">HR Manager</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/258/40388">MAGENTO DEVELOPER</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/258/40387">PHP DEVELOPER(1-3 YEAR EXP)</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/258/40386">SERVER ADMINISTRATOR (1- 4 YEAR ) WITH LINUX EXPERIENCE</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/258/40385">Server Admin- Fresher</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/258/40384">TESTING ENGINEER - TEAM LEAD (4+ Years)</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/258/40383">TESTING ENGINEER (2
- 5 YEARS EXP)</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/258/40382">PHP DEVELOPER (3+ YEAR)</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/258/40381">LARAVEL DEVELOPER (URGENT REQUIREMENT)</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/258/40380">REACT JS DEVELOPER (URGENT REQUIREMENT)</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/258/40379">UI DEVELOPER (0-2 YEAR)</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/258/40378">UI Developer(Minimum 1 year)</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/258/40377">UI DEVELOPER (URGENT REQUIREMENT)</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/258/40376">UI/UX DESIGNER</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/334/40375">Full Stack Developer-Java, Angular, Spring boot, Microservices</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/258/40374">GRAPHIC DESIGNER - TEAM LEAD</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/396/40373">Digital Marketing - Head</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/396/40372">Social Media Marketing Executive</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/258/40371">Content Writer (Urgent requirement)</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/396/40370">Senior React.Js Developer </a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/258/40369">Content Writer - Team Lead (Urgent requirement)</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/258/40368">PERFORMANCE MARKETING (DIGITAL MARKETING)</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/258/40367">Social Media Executive (1 - 3+ Years</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/258/40366">SEO ANALYST (0-1 Year)</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/258/40365">SEO ANALYST</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/258/40364">REACT NATIVE DEVELOPER(URGENT REQUIREMENT)</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/258/40363">Magento Developers(Min 2 year exp)</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/258/40362">ANDROID DEVELOPER (URGENT REQUIREMENT)</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/161/40361">Talent Acquisition Specialist</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/161/40360">Cloud Specialist</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/company-jobs/161/40359">Software Engineer</a></div>, <div class="col-xs-6 col-md-4 mt5"><a href="https://example.com/companies/company/com
I tried the below code but It didn’t work.
res = requests.get(website_url, verify=False)
soup = BeautifulSoup(res.text, 'lxml')
Links = soup.find_all("div", {'class': 'col-xs-6 col-md-4 mt5'},)
url = [tag.get('href') for tag in Links]
2
Answers
You need to use
find
to find thea
tag within thediv
tags and then get the value ofhref
tag from it –you can simply use
BeutigulSoup
and look fora
andhref
: