Question 110080 in Json Question posted in
Our archive of expertly curated questions and answers provides insights and solutions to common problems related to this popular data interchange format. From parsing and manipulating JSON data to integrating it with various programming languages and web services, our archive has got you covered. Start exploring today and take your JSON skills to the next level

Can't get json with axios.get and headers

I am trying to get the joke from https://icanhazdadjoke.com/. This is the code I used

const getDadJoke = async () => {
    const res = await axios.get('https://icanhazdadjoke.com/', {headers: {Accept: 'application/json'}})
    console.log(res.data.joke)
}

getDadJoke()

I expected to get the joke but instead I got the full html page, as if I didn’t specify the headers at all. What am I doing wrong?

Tags:

3

Answers


  1. 0

    Using Axios REST API call which response JSON format.
    If you using API from https://icanhazdadjoke.com/api#authentication
    , you can use Axios.

    Here is example.

    Alternative method.

    You needs to use web scrapping method for this case. Because HTML response from https://icanhazdadjoke.com/.

    This is example how to scrap using puppeteer library in node.js

    Demo code

    Save as get-joke.js file.

    const puppeteer = require("puppeteer");

async function getJoke() {
    try {
        const browser = await puppeteer.launch();
        const page = await browser.newPage();
        await page.goto('https://icanhazdadjoke.com/');

        const joke = await page.evaluate(() => {
            const jokes = Array.from(document.querySelectorAll('p[class="subtitle"]'))
            return jokes[0].innerText;
        });
        
        await browser.close();
        return Promise.resolve(joke);

    } catch (error) {
        return Promise.reject(error);
    }
}

getJoke()
    .then((joke) => {
        console.log(joke);
    })

    Selector

    Main Idea to use DOM tree selector

    In the Chrome’s DevTool (by pressing F12), shows HTML DOM tree structures.

    <p> tag has class name is subtitle

    document.querySelectorAll('p[class="subtitle"]')

    enter image description here

    Install dependency and run it

    npm install puppeteer
node get-joke.js

    Result

    You can get the joke from that web site.

    enter image description here

    Login or Signup to reply.
  2. 0

    The HTML page you’re getting is a 503 response from Cloudflare.

    As per the API documentation

    Custom user agent

    If you intend on using the icanhazdadjoke.com API we kindly ask that you set a custom User-Agent header for all requests.

    My guess is they have a Cloudflare Browser Integrity Check configured that’s triggering for the default Node / Axios user-agent.

    Setting a custom user-agent appears to get around this…

    const getDadJoke = async () => {
  try {
    const res = await axios.get("https://icanhazdadjoke.com/", {
      headers: {
        accept: "application/json",
        "user-agent": "My Node and Axios app", // use something better than this
      },
    });
    console.log(res.data.joke);
  } catch (err) {
    console.error(err.response?.data, err.toJSON());
  }
};

    Given how unreliable Axios releases have been since v1.0.0, I highly recommend you switch to something else. The Fetch API is available natively in Node since v18

    const getDadJoke = async () => {
  try {
    const res = await fetch("https://icanhazdadjoke.com/", {
      headers: {
        accept: "application/json",
        "user-agent": "My Node and Fetch app", // use something better than this
      },
    });

    if (!res.ok) {
      const err = new Error(`${res.status} ${res.statusText}`);
      err.text = await res.text();
      throw err;
    }
    console.log((await res.json()).joke);
  } catch (err) {
    console.error(err, err.text);
  }
};
    Login or Signup to reply.
  3. 0

    If you look at the API documentation for icanhazdadjoke.com, there is a section titled "Custom user agent." In that section, they explain how they want any requests to have a User Agent header. If you use Axios in a browser context, the User Agent is set for you by your browser. But I’m going to go out on a limb and say that you are running this code via Node, in which case, you may manually need to set the User Agent header, like so:

    const getDadJoke = async () => {
    const res = await axios.get(
        'https://icanhazdadjoke.com/',
        {
            headers:
            {
                'Accept': 'application/json',
                'User-Agent': 'my URL, email or whatever'
            }
        }
    )
    console.log(res.data.joke)
}

getDadJoke()

    The docs say what they want you to put for the User Agent, but I think it would honestly work if there were any User Agent field at all.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search