skip to Main Content

I have valious formats of incoming JSON from 3rd parties that I need to process in unified manner. Considering the data structure is not normalized, I could not apply unified logic of data processing. To achieve that I’m seeking for data formatting into single format before processing.

Source 1:

{
  "fieldName": "status",
  "oldValue": "Open",
  "newValue": "Closed"
}

Source 2:

{
  "fieldName": "status",
  "oldValue": {
    "name": "Open"
  },
  "newValue": {
    "name": "Closed"
  }
}

Source 3:

{
   "fieldName": "reason",
   "oldValue": null,
   "newValue": {
      "name": "Requirements changed"
   }
}

Source 4:

{
   "fieldName": "advertise",
   "oldValue": true,
   "newValue": false
}

In result transformation I would like to have the following (combined for all sources):

{
   "status"|"reason"|"advertise": {
      "oldValue": {
         "value": "Open"|null|true
      },
      "newValue": {
         "value": "Closed"|"Requirements changed"|false
      }
   }
}

I’m struggling with various combinations of shift and default but could not achieve the transformation. I tried to build JOLT specification that makes this transformation (see example below), however I’m unable to put different field structure (simple value vs object) into unified format.

[
  {
    "operation": "shift",
    "spec": {
      "fieldName": {
        "*": {
          "@(2,oldValue)": "&1.oldValue",
          "@(2,newValue)": "&1.newValue"
        }
      }
    }
  }
]

Sample of code (Java) to reproduce the problem

package org.example;

import com.bazaarvoice.jolt.Chainr;
import com.bazaarvoice.jolt.JsonUtils;

import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.util.List;

public class Main {
    public static void main(String[] args) throws IOException {
        var spec = read("src/main/resources/spec.json");
        var chainr = Chainr.fromSpec(JsonUtils.jsonToList(spec));

        var example1 = read("src/main/resources/example1.json");
        var example2 = read("src/main/resources/example2.json");
        var example3 = read("src/main/resources/example3.json");
        var example4 = read("src/main/resources/example4.json");

        for (String source: List.of(example1, example2, example3, example4)) {
            System.out.println("===nSource n" + source);
            var transformedSource = chainr.transform(JsonUtils.jsonToMap(source));

            System.out.println("nAfter transformation n" +
                    JsonUtils.toPrettyJsonString(transformedSource));
        }
    }

    private static String read(String path) throws IOException {
        return new String(Files.readAllBytes(Paths.get(path)));
    }
}

You also need the following dependencies to achieve data transformation

        <dependency>
            <groupId>com.bazaarvoice.jolt</groupId>
            <artifactId>jolt-core</artifactId>
            <version>0.1.8</version>
        </dependency>
        <dependency>
            <groupId>com.bazaarvoice.jolt</groupId>
            <artifactId>json-utils</artifactId>
            <version>0.1.8</version>
        </dependency>

Any thoughts how to make it possible?

Thanks in advance!

2

Answers


  1. The following spec should work for all the scenarios:

    [
     
      {
        "operation": "shift",
        "spec": {
          "oldValue": {
            "name": {
              "@": "@(3,fieldName).oldValue.value"
            },
            "*": {
              "@1": "@(3,fieldName).oldValue.value"
            }
          },
          "newValue": {
            "name": {
              "@": "@(3,fieldName).newValue.value"
            },
            "*": {
              "@1": "@(3,fieldName).newValue.value"
            }
          }
        }
      },
      {
        "operation": "default",
        "spec": {
          "*": {
            "oldValue": {
              "value": null
            },
            "newValue": {
              "value": null
            }
          }
        }
      }
    
    ]
    
    Login or Signup to reply.
  2. More straightforward option would be

    [
      {//handles null cases only through prefixing by the ~ operator 
        "operation": "modify-overwrite-beta",
        "spec": {
          "~*V*": "null"//convert the value into "null" if it's null
        }
      },
      {
        "operation": "shift",
        "spec": {//separate the cases for attributes of "name" vs. the others  
          "*V*": { //represents *Values
            "name": { "@": "@3,fieldName.&2.&" },//stands for the objects 
            "*": { "@1": "@3,fieldName.&2.v&(2,2)" }//stands for the plain attributes
          }
        }
      }
    ]
    
    Login or Signup to reply.
Please signup or login to give your own answer.
Back To Top
Search