Question 110478 in Json Question posted in
Our archive of expertly curated questions and answers provides insights and solutions to common problems related to this popular data interchange format. From parsing and manipulating JSON data to integrating it with various programming languages and web services, our archive has got you covered. Start exploring today and take your JSON skills to the next level

Java: Protobuf byte to Json string to Pojo fast

I am receiving messages in protobuf format. I need to convert it to json format fast as all my business logic is written to handle json based POJO objects.

byte[] request = ..; // msg received

// convert to intermediate POJO
AdxOpenRtb.BidRequest bidRequestProto = AdxOpenRtb.BidRequest.parseFrom(request, reg);

// convert intermediate POJO to json string.
// THIS STEP IS VERY SLOW
Printer printer = JsonFormat.printer().printingEnumsAsInts().omittingInsignificantWhitespace();
String jsonBody = printer.print(bidRequestProto);

// convert json string to final POJO format
BidRequest bidRequest = super.parse(jsonBody.getBytes());

Proto object to json conversion step is very slow. Is there any faster approach for it?

can i reuse printer object? is it thread-safe?

Note: This POJO class (AdxOpenRtb.BidRequest & BidRequest) is very complex having many hierarchy and fields but contains similar data with slightly different fields name and data types.

Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    I end up using MapStruct as suggested by some of you (@M.Deinum).

    new code:

    byte[] request = ..; // msg received

// convert to intermediate POJO
AdxOpenRtb.BidRequest bidRequestProto = AdxOpenRtb.BidRequest.parseFrom(request, reg);

// direct conversion from protobuf Pojo to my custom Pojo
BidRequest bidRequest = BidRequestMapper.INSTANCE.adxOpenRtbToBidRequest(bidRequestProto);

    Code snippet of BidRequestMapper:

    @Mapper(
    collectionMappingStrategy = CollectionMappingStrategy.ADDER_PREFERRED, nullValueCheckStrategy = NullValueCheckStrategy.ALWAYS,
    unmappedSourcePolicy = ReportingPolicy.WARN, unmappedTargetPolicy = ReportingPolicy.WARN)
@DecoratedWith(BidRequestMapperDecorator.class)
public abstract class BidRequestMapper {

  public static final BidRequestMapper INSTANCE = Mappers.getMapper(BidRequestMapper.class);

  @Mapping(source = "impList", target = "imp")
  @Mapping(target = "impOverride", ignore = true)
  @Mapping(target = "ext", ignore = true)
  public abstract BidRequest adxOpenRtbToBidRequest(AdxOpenRtb.BidRequest adxOpenRtb);
...
...
}

// manage proto extensions
abstract class BidRequestMapperDecorator extends BidRequestMapper {
  private final BidRequestMapper delegate;

  BidRequestMapperDecorator(BidRequestMapper delegate) {
    this.delegate = delegate;
  }

  @Override
  public BidRequest adxOpenRtbToBidRequest(AdxOpenRtb.BidRequest bidRequestProto) {
    // Covert protobuf msg to basic bid request object
    BidRequest bidRequest = delegate.adxOpenRtbToBidRequest(bidRequestProto);
...
...
  }
}

    The new approach is 20-50x faster in my local test environment.

    It's worth mentioning that MapStruct is an annotation processor which makes it much faster than other similar libraries which use reflection and it also has very good support for customization.


  2. 0

    I ran into some performance issues as well and ended up writing the QuickBuffers library. It generates dedicated JSON serialization methods (i.e. no reflection) and should give you a 10-30x speedup. It can be used side-by-side with Google’s implementation. The code should look something like this:

        // Initialization (objects can be reused if desired)
    AdxOpenRtb.BidRequest bidRequestProto = AdxOpenRtb.BidRequest.newInstance();
    ProtoSource protoSource = ProtoSource.newArraySource();
    JsonSink jsonSink = JsonSink.newInstance().setWriteEnumsAsInts(true);

    // Convert Protobuf to JSON
    bidRequestProto.clearQuick() // or ::parseFrom if you want a new object
            .mergeFrom(protoSource.setInput(request))
            .writeTo(jsonSink.clear());

    // Use the raw json bytes
    RepeatedByte jsonBytes = jsonSink.getBytes();

    JsonSinkBenchmark has some sample code for replacing the built-in JSON encoder with more battle-tested Gson/Jackson backends.

    Edit: if you’re doing this within a single process and are worried about performance, you’re better off writing or generating code to convert the Java objects directly. JSON is not a very efficient format to go through.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search