Json Question posted in
Our archive of expertly curated questions and answers provides insights and solutions to common problems related to this popular data interchange format. From parsing and manipulating JSON data to integrating it with various programming languages and web services, our archive has got you covered. Start exploring today and take your JSON skills to the next level

Deserialize json array into existing Vec

I have some function, here called request, that returns a JSON string with some data. For example, an array with variable size. I can deserialize the JSON string into a Vec just fine with serde_json, but I have to call the request many times. I would like instead to reuse the Vec already created, instead of dropping and creating a new Vec every time deserialization happens.

Is there a way to deserialize a JSON string into an existing Vec? I had a look at the visitor and at this example, but I am not sure which I would have to implement. Either a custom visitor or a deserializer that wraps the serde_json::Deserializer and has a reference to a Vec, or perhaps something else entirely.

Here is some code with an example. Thanks.

use serde_json;

fn request(i: usize) -> String {
    let mut v = vec![0; i];

    for n in 1..i {
        v[n] = n
    }

    serde_json::to_string(&v).unwrap()
}

fn main() {
    let mut bigvec = serde_json::from_str::<Vec<i32>>(&request(100)).unwrap();
    println!("{:?}", &bigvec);

    // - now we will rerun the request many times and we would like to reuse our vec

    let many_iterations = 10;
    for n in 1..many_iterations {
        bigvec.clear();

        let resp = request(n); // we receive a new request with a vec with different length

        // - we would like something like the below, which does not exist
        serde_json::from_str_into::<Vec<i32>>(&resp, &mut bigvec);
    }
}

Playground

Tags:

2

Answers


  1. 0

    We can’t fully deserialize everything in place because serde is designed around returning the value as we deserialize it. However, it you just want to deserialize into a single Vec, we can make a visitor to handle that situation. Here is roughly what that would look like.

    use serde::de::{DeserializeOwned, SeqAccess, Visitor};
use serde::{Deserialize, Deserializer};
use std::fmt::{self, Formatter};

struct InPlaceVisitor<'a, T> {
    buffer: &'a mut Vec<T>,
}

impl<'de, 'a: 'de, T> Visitor<'de> for InPlaceVisitor<'a, T>
where
    T: Deserialize<'de>,
{
    /// We are deserializing in place so we won't have any output
    type Value = ();

    /// Every visitor needs to implement this function to add some
    /// context to error messages. This will be called if any of the
    /// `visit_*` functions we didn't implement get called.
    fn expecting(&self, f: &mut Formatter<'_>) -> fmt::Result {
        write!(f, "sequence of values")
    }

    /// Handle the case where the deserializer finds a sequence of
    /// values.
    fn visit_seq<A>(self, mut seq: A) -> Result<Self::Value, A::Error>
    where
        A: SeqAccess<'de>,
    {
        // Clear the buffer before we begin
        self.buffer.clear();

        // If the deserializer knows the number of items in the
        // sequence we can ensure we have enough capacity before
        // starting. However, this is generally only the case for
        // formats that prefix sequences with their lengths.
        if let Some(hint) = seq.size_hint() {
            self.buffer.reserve(hint);
        }

        // Now we can just read the values and add them to the
        // buffer. This will call serde::Deserialize::deserialize for 
        // each element.
        while let Some(value) = seq.next_element()? {
            self.buffer.push(value);
        }

        Ok(())
    }
}

    Now, one of the tricky parts of using this is that we need to construct our visitor with the buffer we want to push to. This is tricky that means we will not have access to our buffer within serde::Deserialize::deserialize. As a result, we must construct the deserializer ourselves. Not all serde formats expose their Serializer and Deserializer types so this solution won’t be possible for all data formats. Luckily serde_json does expose its deserializer, so we can implement a general function to use our visitor.

    pub fn deserialize_json_in_place<'de, 'a: 'de, T, R>(
    reader: R,
    buffer: &'a mut Vec<T>,
) -> serde_json::Result<()>
where
    T: Deserialize<'de>,
    R: serde_json::de::Read<'de>,
{
    let mut deserializer = serde_json::de::Deserializer::new(reader);

    let visitor = InPlaceVisitor { buffer };

    // We tell the deserializer what we are hoping to find, then the
    // deserializer calls the `visit_*` function corresponding to the
    // type it actually encountered.
    deserializer.deserialize_seq(visitor)
}

    With this, we are all set. However for convenience, you may still want to make some simple wrapper functions to handle creating the reader.

    pub fn from_str_into<T>(x: &str, buffer: &mut Vec<T>) -> serde_json::Result<()>
where
    T: DeserializeOwned,
{
    let reader = serde_json::de::StrRead::new(x);
    deserialize_json_in_place(reader, buffer)
}

    Rust Playground

    Login or Signup to reply.
  2. 0

    There is a hidden deserialize_in_place method on the Deserialize trait that can support this1. By default it would simply create a new value and assign it to the existing one, but Vec‘s impl does exactly what you want.

    So you just need to call it manually with the serde-json deserializer:

    use serde::Deserialize;
use serde_json::Deserializer;

let mut deserializer = Deserializer::from_str(&resp);
Deserialize::deserialize_in_place(&mut deserializer, &mut bigvec).unwrap();

    1. This is not a private API. The documentation says "This method is stable and an official public API, but hidden from the documentation because it is almost never what newbies are looking for."

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search