Question 224423 in Html Question posted in
Some basic documentation can be found here.

Html – How do I extract a table from a website that needs to be searched for first

So I’m trying to extract a table from this website https://careersportal.ie/courses/simple_search.php . I tried using
courses = pd.read_html('https://careersportal.ie/courses/simple_search.php')
but I’m getting some sort of Import Error saying I need to install html5lib, which I’ve install with pip but still get the same issue.

I tried using the built-in pandas methods pd.read_csv() and pd.read_html

Tags:

2

Answers


  1. 0

    You’re trying to extract a table from a website using Python and pandas,
    Before you can extract data from a website, you need to understand its structure and how the table you want to extract is represented in the HTML.

    1. Parse HTML Using BeautifulSoup
      use BeautifulSoup to parse the HTML content and locate the table you want.
    2. Extract Data from the Table
    Login or Signup to reply.
  2. 0

    To get the table into a pandas DataFrame you can use next example:

    import pandas as pd
import requests

api_url = "https://cc.careersportal.ie/courses/course_search_ajax.php"

params = {
    "p": "1",
    "ord": "2",
    "dir": "0",
    "txt": "",
    "default": "false",
    "college_filter": "",
}

all_dfs = []
for params["p"] in range(1, 4):   # <-- increase number of pages here
    data = requests.get(api_url, params=params).json()
    all_dfs.append(pd.DataFrame(data["rows"]))

df = pd.concat(all_dfs, ignore_index=True)
print(df.tail())

    Prints:

            id    tag            code  cdirect  client_id     dur                                                   ttl                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            overview   lnk                                        col col_id col_lnk  pts pts_cpy  pts_new sym1 sym2 pts_dif   qqi         awd  course_category                                                                                                                                                                                                                                                                                                                                                                                                                                                 opportunities interests
295  20773  False  5M3807 CI061FO     True        147  1 Year                           Applied Science - Forensics                                                                                                                                                                                                             <p>Science is defined as any science which is used as evidence in a court of law. This course will provide participants with a broad knowledge of the core sciences and an understanding of the Irish legal system.</p>rn<p>Students explore the fundamentals of forensic science, evidence collection and analysis. Students will gain skills in fingerprinting, exposing and lifting prints, analysing hair and fibres, casting foot and tyre prints, blood spatter analysis and DNA profiling.</p>rn<p>Participants will develop useful skills through hands-on laboratory work as well as an appreciation of the concepts particular to forensics. Students will also develop skills in problem solving, organisation, team work, effective communication and the use of new technology, an important emphasis is placed on working safely.</p>  None                            Cavan Institute    216    None  NaN            False              None  True  level5.jpg                1                                                                                                                                                                                                                              Graduates of this course have a strong foundation in the skills required to obtain positions in laboratories or other science related areas such as biotechnology, pharmaceutical science, chemical and biochemical engineering.       [I]
296  21746  False          5M3807    False          0  1 Year               Applied Science - Laboratory Techniques                                                                                                                                                                                                                                                                                                         Ireland&rsquo;s pharmaceutical industry produces a wide range of products and services, from research and development of new medicines to the manufacturing and marketing of new medicines for humans and animals. Personnel with scientific knowledge and laboratory analytical skills are essential for this industry sector. <br /><br />The main focus of this course will be to enhance your scientific knowledge and laboratory skills in a range of applied science fields. This one-year full time course has assisted the majority of our past graduates in progressing to third level courses in Universities and IT&rsquo;s. It is also suitable for individuals who wish to enhance their chances of employment in the pharmaceutical industry.  None  Cork College of FET - Tramore Road Campus    714    None  NaN            False              None  True  level5.jpg                1                                                                                                                                                                                                               The pharmaceutical industry is one of Ireland&rsquo;s largest employers and have a large potential for employment. Job opportunities for students who complete this course include: Laboratory assistants, Operators pharmaceutical industries.       [I]
297  21392  False          5M3807     True        258  1 Year  Applied Science - Laboratory Techniques (Clash Road)                                                                                                                                                                                                                                                                                                                                                                                                                                                         This programme provides an introduction to the world of science and covers a variety of subjects related to laboratory sciences. Upon successful completion of this course, you may be eligible to apply through the CAO and/or the Higher Education Links Scheme for entry to a range of higher certificate and degree programmes at Institutes of Technology and Universities. There are progression options at Levels 6, 7 and 8 in a wide range of areas including general science, biotechnology, forensic science, pharmaceutical science, sports science, food science and many other related areas.  None                              Kerry College   2322    None  NaN            False              None  True  level5.jpg                1  This course prepares learners for careers typically found in the following sectors: <br />rn<ul>rn<li>Chemical Industry</li>rn<li>Biopharma-Pharmachem</li>rn<li>Cosmetic Science</li>rn<li>Food and Beverages</li>rn<li>Physical and Mathematical Sciences</li>rn<li>Medical Devices, Education and Research</li>rn<li>Environmental Science</li>rn<li>Human Biology and Health</li>rn<li>Science Journalism and Business</li>rn</ul>       [I]
298  22250  False    5M3807 CI061     True        147  1 Year                Applied Science - Nutrition and Health  <p>Nutrition is the branch of science that deals with nutrients and how we obtain them and their effects in the body. Good nutritional choices are important for good health and preventing disease. Knowledge of nutrition allows an individual to make informed choices about what foods they or their families should eat. An understanding of how food is produced and how it affects the body is important for many career paths. Nutrition is a growth area with jobs in a variety of fields for example food science, food production, sports science, health and fitness industries.</p>rn<p>This course will prepare the learner for direct employment or for progression to third level and is accepted for entry to most science, food science, nutrition or health related courses in Ireland.</p>rn<p>This course also includes maths, biology, chemistry, physics and IT skills. These subjects are common to all first year science qualifications and give students an advantage over students entering directly from leaving certificate.</p>  None                            Cavan Institute    216    None  NaN            False              None  True  level5.jpg                1                                                                                                                                                                                                Graduates of this course have a strong foundation in the skills required to obtain positions in laboratories or other science/health related areas such as biotechnology, pharmaceutical science, sports science, food science and general science industries.    [I, S]
299  20568  False  5M3807 CI061PH     True        147  1 Year                    Applied Science - Pharmacy Studies                                                                                                                                                                                                                                                                                                                                                                             Pharmacies also sell products such as baby care accessories, cosmetics, perfumes and health supplements. Sales of these products are becoming an increasingly important part of a pharmacys income and graduates of this course will be well placed to recommend and sell such products, having studied modules in customer service and beauty therapy as part of this course. As a pharmacy assistant, you will work as part of a team of health professionals in order to help customers achieve the best possible health outcomes. Students of this course may also be offered the opportunity to complete the Medicine Counter Assistant award from the Irish Pharmacy Union (IPU).  None                            Cavan Institute    216    None  NaN            False              None  True  level5.jpg                1                                                                               Students who complete this course will graduate with a good mix of science, retail and customer service knowledge and skills. Graduates may obtain employment in local pharmacies, assisting the pharmacist to provide a complete care service to the community including health care advice, beauty and cosmetic tips, good customer service, retail display and sales skills.       [I]
    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search