Question 189620 in Json Question posted in
Our archive of expertly curated questions and answers provides insights and solutions to common problems related to this popular data interchange format. From parsing and manipulating JSON data to integrating it with various programming languages and web services, our archive has got you covered. Start exploring today and take your JSON skills to the next level

I have several JSON's of the same structure. They have object that is an array of objects . How to I append these arrays into one array?

What I want to do:

I want to send several GET-requests to this URL:
https://catalog.wb.ru/brands/m/catalog?page=1&limit=300&brand=5786&dest=-1257786&sort=pricedown
Then gather all the data inside of a "product" object. The value of key "PAGE" is autoincrementing to get data from all pages.

Actually I’m not quite sure that I really need to make up one JSON to send it to front-end. Maybe it’s better to send different requests as I get new responses in a for loop?

What I did:

Made correct structures. With single request everything is working fine.

Created requestBodyBytes []byte and ProductsBytes []byte in order to append them with []bytes from ioutil.ReadAll.
Printing length of requestBodyBytes I see that it expands with every request but after I Unmarshal it I see empty struct in the output.

I understand that it happens because every single request I get new JSON of type Response. But what if I need a slice of Product structs composed of "product" objects from several JSON’s of type Response?

Note: needed to init requestBodyBytes inside of for loop to use it to stop sending requests because the server gives 200 code and empty JSON when there’s no information on a page.

Thank you in advance!

const URL = "https://catalog.wb.ru/brands/m/catalog?page=%d&limit=300&brand=5786&dest=-1257786&sort=pricedown"

type Response struct {
    Data struct {
        Products []Product `json:"products"`
    } `json:"data"`
}

type Product struct {
    ID     int     `json:"id"`
    Name   string  `json:"name"`
    Price  int     `json:"priceU"`
    Rating float32 `json:"reviewRating"`
    Sale   int     `json:"sale"`
    New    bool    `json:"isNew"`
}

func main() {
    var response Response
    var products Response //Also tried to make it []Response
    var ProductsBytes []byte

    for i := 1; ; i++ {
        resp, err := http.Get(fmt.Sprintf(URL, i))
        if err != nil {
            fmt.Printf("#1 Error: %s", err)
        }
        defer resp.Body.Close()

        bytes, err := ioutil.ReadAll(resp.Body)
        var requestBodyBytes []byte
        requestBodyBytes = append(requestBodyBytes, bytes...)
        ProductsBytes = append(ProductsBytes, bytes...)

        json.Unmarshal(requestBodyBytes, &response)

        fmt.Println(resp.Status)
        fmt.Printf("nSlice from page #%dnLength of bytes: %dn", i, len(bytes))
        fmt.Printf("Length of finalResult: %dn", len(requestBodyBytes))
        if len(response.Data.Products) == 0 {
            fmt.Println("There's no more data")
            break
        }
    }

    json.Unmarshal(ProductsBytes, &products)

    fmt.Println(response)
    fmt.Println(products)
    fmt.Println(len(products))
}
Tags:

2

Answers


  1. 0

    Instead of unmarshaling the reponse with []byte, we can unmarshal it with the Response struct.

    Here is the updated code

    package main

import (
    "encoding/json"
    "fmt"
    "io/ioutil"
    "net/http"
)

const URL = "https://catalog.wb.ru/brands/m/catalog?page=%d&limit=300&brand=5786&dest=-1257786&sort=pricedown"

type Response struct {
    Data struct {
        Products []Product `json:"products"`
    } `json:"data"`
}

type Product struct {
    ID     int     `json:"id"`
    Name   string  `json:"name"`
    Price  int     `json:"priceU"`
    Rating float32 `json:"reviewRating"`
    Sale   int     `json:"sale"`
    New    bool    `json:"isNew"`
}

func main() {
    var products = []Product{}

    for i := 1; ; i++ {
        resp, err := http.Get(fmt.Sprintf(URL, i))
        if err != nil {
            fmt.Printf("#1 Error: %s", err)
        }
        defer resp.Body.Close()

        bytes, err := ioutil.ReadAll(resp.Body)
        if err != nil {
            fmt.Printf("#2 Error: %s", err)
        }

        // The response has defined as `Response` type and unmarshaling
        var response Response
        err = json.Unmarshal(bytes, &response)
        if err != nil {
            fmt.Printf("#3 Error: %s", err)
            break
        }

        fmt.Printf("nSlice from page #%dnLength of bytes: %dn", i, len(bytes))
        fmt.Printf("Length of finalResult: %dn", len(response.Data.Products))

        if len(response.Data.Products) == 0 {
            fmt.Println("There's no more data")
            break
        }

        // if there has products then append into the Products slice
        products = append(products, response.Data.Products...)
    }

    fmt.Println("products:", products)
    fmt.Println("Total products count:", len(products))

}
    Login or Signup to reply.
  2. 0

    There is no reason to collect all the raw response bytes. Just unmarshal each response individually and append the products for each page to some slice holding all of the products. Also, calling defer resp.Body.Close() in a loop is probably not what you want. The deferred statement executes only after the loop finishes, so connections cannot be re-used for the requests. Extracting the loop body into its own function makes this much cleaner:

    package main

import (
    "encoding/json"
    "errors"
    "fmt"
    "log"
    "net/http"
)

const URL = "https://catalog.wb.ru/brands/m/catalog?page=%d&limit=300&brand=5786&dest=-1257786&sort=pricedown"

type Response struct {
    Data struct {
        Products []Product `json:"products"`
    } `json:"data"`
}

type Product struct {
    ID     int     `json:"id"`
    Name   string  `json:"name"`
    Price  int     `json:"priceU"`
    Rating float32 `json:"reviewRating"`
    Sale   int     `json:"sale"`
    New    bool    `json:"isNew"`
}

func main() {
    var allProducts []Product

    for i := 1; ; i++ {
        page, err := fetchPage(i)
        if err != nil {
            log.Fatal(err) // TODO
        }

        allProducts = append(allProducts, page...)

        if len(page) == 0 {
            break
        }
    }

    fmt.Println(allProducts)
    fmt.Println(len(allProducts))
}

func fetchPage(i int) ([]Product, error) {
    resp, err := http.Get(fmt.Sprintf(URL, i))
    if err != nil {
        return nil, err
    }
    defer resp.Body.Close()

    if resp.StatusCode != 200 {
        return nil, errors.New(resp.Status)
    }

    var response Response
    err = json.NewDecoder(resp.Body).Decode(&response)
    if err != nil {
        return nil, err
    }

    return response.Data.Products, nil
}
    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search