skip to Main Content

I am doing a spring batch project in which I have to read and modify several TXT files and write them in a new file in JSON format. I created my steps for each file, they are written one after the other. However, the last two have a link, the cities, and history files. Some cities have a history, and it would be necessary for each city if a history has the same postal code as it, then its history is rewritten following this one. And I’m stuck a little, I don’t know if I should rewrite a step for each or if I should do both in the same step. At the moment, I have this:


  @Bean
  public Job runjob(JobRepository jobRepository, PlatformTransactionManager txManager) throws Exception{
    return new JobBuilder("runJob", jobRepository)
        .start(stepCountry(jobRepository,txManager))
        .next(stepRegion(jobRepository,txManager))
        .next(stepDepartment(jobRepository,txManager))
        .next(stepCity(jobRepository,txManager))
        .next(stepHistorique(jobRepository,txManager))
        .build(); 
}

for the moment, all my @Beans are in one class config

  @Bean
  public Step stepCity(JobRepository jobRepository, PlatformTransactionManager txManager)throws Exception{
    return new StepBuilder("stepCity")
            .repository(jobRepository)
            .<City, City>chunk(25)
            .reader(readerCity())
            .processor(processorCity())
            .writer(writerCity())
            .transactionManager(txManager)
            .build();
  }

  @Bean
  public Step stepHistorique(JobRepository jobRepository, PlatformTransactionManager txManager)throws Exception{
    return new StepBuilder("stepHistorique")
            .repository(jobRepository)
            .<Historique, Historique>chunk(25)
            .reader(readerHistorique())
            .processor(processorHistorique())
            .writer(writerHistorique())
            .transactionManager(txManager)
            .build();
  }

I did not put what concerns my processor and my other files because it is not important for what I want to do currently


@Bean
  public ItemReader<City> readerCity() throws Exception{
    FlatFileItemReader<City> readerCity = new FlatFileItemReader<City>();
    readerCity.setLineMapper(new DefaultLineMapper() {{
      setLineTokenizer(new DelimitedLineTokenizer() {{
        setNames(new String[]{"typecom","com","reg","dep","arr","tncc","ncc","nccenr","libelle","can","comparent"});
      }});
      setFieldSetMapper(new BeanWrapperFieldSetMapper<City>() {{
        setTargetType(City.class);
      }});
    }});
    readerCity.setResource(new ClassPathResource("documents/Cities.txt"));
    return readerCity;
  }

  @Bean
  public ItemReader readerHistorique() throws Exception{
    FlatFileItemReader readerHistorique = new FlatFileItemReader();
    readerHistorique.setLineMapper(new DefaultLineMapper() {{
      setLineTokenizer(new DelimitedLineTokenizer() {{
        setNames(new String[]{"mod","date_eff","typecom_av","com_av","tncc_av","ncc_av","nccenr_av","libelle_av","typecom_ap","com_ap"
            ,"tncc_ap","ncc_ap","nccenr_ap", "libelle_ap"});
      }});
      setFieldSetMapper(new BeanWrapperFieldSetMapper() {{
        setTargetType(Historique.class);
      }});
    }});
    readerHistorique.setResource(new ClassPathResource("documents/HistoriqueCities.txt"));
    System.out.println(readerHistorique);
    return readerHistorique;
  }

I think it is necessary for everyone to have an ItemReader

  @Bean
  public ItemWriter<City> writerCity() {
    JsonFileItemWriter<City> writerCity = new JsonFileItemWriter<>(
        new FileSystemResource("src/main/java/output/Insee.json"),
        new JsonObjectMarshaller<City>() {
          @Override
          public String marshal(City object) {
            try {
              ObjectMapper mapper = new ObjectMapper();
              mapper.configure(SerializationFeature.INDENT_OUTPUT, true);
              return mapper.writeValueAsString(object);
            } catch (JsonProcessingException e) {
              e.printStackTrace();
              return null;
            }
          }
        }) {
      @Override
      public void open(ExecutionContext executionContext)
          throws ItemStreamException {
        super.open(executionContext);
        try {
          getOutputState().write(",{Cities: }[");
        }
        catch(IOException e) {
          throw new RuntimeException(e);
        }
      }
    };
    writerCity.setResource(new FileSystemResource("src/main/java/output/Insee.json"));
    writerCity.setAppendAllowed(true);
    writerCity.setJsonObjectMarshaller(new JsonObjectMarshaller<City>() {
      @Override
      public String marshal(City item) {
        Map<String, Object> map = new LinkedHashMap<>();
        map.put("status", item.getTypecom());
        map.put("insee", item.getCom());
        map.put("parent", item.getComparent());
        map.put("country", item.getCountry());
        map.put("dept", item.getDep());

        return map.toString();
      }
    });
    return writerCity;
  }

@Bean
  public ItemWriter<Historique> writerHistorique() {
    JsonFileItemWriter<Historique> writerHistorique = new JsonFileItemWriter<>(
        new FileSystemResource("src/main/java/output/Insee.json"),
        new JsonObjectMarshaller<Historique>() {
          @Override
          public String marshal(Historique object) {
            try {
              ObjectMapper mapper = new ObjectMapper();
              mapper.configure(SerializationFeature.INDENT_OUTPUT, true);
              return mapper.writeValueAsString(object);
            } catch (JsonProcessingException e) {
              e.printStackTrace();
              return null;
            }
          }
        }) {
      @Override
      public void open(ExecutionContext executionContext)
          throws ItemStreamException {
        super.open(executionContext);
        try {
          getOutputState().write(",{Historique: } [");
        }
        catch(IOException e) {
          throw new RuntimeException(e);
        }
      }
    };
    writerHistorique.setResource(new FileSystemResource("src/main/java/output/Insee.json"));
    writerHistorique.setAppendAllowed(true);
    writerHistorique.setJsonObjectMarshaller(new JsonObjectMarshaller<Historique>() {
      @Override
      public String marshal(Historique item) {
        Map<String, Object> map = new LinkedHashMap<>();
        map.put("parent", item.getParent());
        map.put("typecom_av", item.getTypecom_av());
        map.put("mod", item.getMod());
        map.put("nccenr_av", item.getLibelle_av());
        map.put("nccenr_av", item.getLibelle_ap());
        map.put("date_eff", item.getDate_eff());
        return map.toString();
      }
    });
    return writerHistorique;
  }

I don’t know if they should each have their own ItemWriter or if I should write them in just one. City ​​and history each have their class

2

Answers


  1. Chosen as BEST ANSWER

    Yes of course. I have to read my cities.txt file which has this structure:

    "typecom","com","reg","dep","arr","tncc","ncc","nccenr","libelle","can","comparent"
    COM,"01001","84","01","012","5",ABERGEMENT CLEMENCIAT,Abergement-Clémenciat,L'Abergement-Clémenciat,"0108",
    COM,"01002","84","01","011","5",ABERGEMENT DE VAREY,Abergement-de-Varey,L'Abergement-de-Varey,"0101",
    

    then I have to read my history.txt file which has this structure:

    mod,date_eff,typecom_av,com_av,tncc_av,ncc_av,nccenr_av,libelle_av,typecom_ap,com_ap,tncc_ap,ncc_ap,nccenr_ap,libelle_ap
    

    32,2019-03-01,COM,45287,0,SAINT LOUP DE GONOIS,Saint-Loup-de-Gonois,Saint-Loup-de-Gonois,COMD,45287,0,SAINT LOUP DE GONOIS,Saint-Loup-de-Gonois,Saint-Loup-de-Gonois 32,2019-03-01,COM,45287,0,SAINT LOUP DE GONOIS,Saint-Loup-de-Gonois,Saint-Loup-de-Gonois,COM,45307,3,SELLE SUR LE BIED,Selle-sur-le-Bied,La Selle-sur-le-Bied.

    and I have to rewrite them in json format so that each city if it has a "com" number identical to the "com_av" history then following this city its history appears. like that :

     {
        "status" : "DELEGATE",
        "insee" : "50417",
        "parent" : "50417",
        "country" : "fr",
        "dept" : "",
        "historics" : [ {
          "parent" : "50417",
          "typeModeNumber" : "32",
          "typemod" : "COM",
          "label" : "Quettehou",
          "oldLabel" : "Morsalines",
          "dateEffet" : "2019-01-01"
        }    
    

    I skip the part where I process the data to display only what I need. but for the moment in the code that I gave I read the two files and when I write them I display all the cities and then all the histories. and that's not what I want.


  2. Did you try implement your own ItemReader that fetch both files to the record that you need be written?

    Login or Signup to reply.
Please signup or login to give your own answer.
Back To Top
Search