I am doing a spring batch project in which I have to read and modify several TXT files and write them in a new file in JSON format. I created my steps for each file, they are written one after the other. However, the last two have a link, the cities, and history files. Some cities have a history, and it would be necessary for each city if a history has the same postal code as it, then its history is rewritten following this one. And I’m stuck a little, I don’t know if I should rewrite a step for each or if I should do both in the same step. At the moment, I have this:
@Bean
public Job runjob(JobRepository jobRepository, PlatformTransactionManager txManager) throws Exception{
return new JobBuilder("runJob", jobRepository)
.start(stepCountry(jobRepository,txManager))
.next(stepRegion(jobRepository,txManager))
.next(stepDepartment(jobRepository,txManager))
.next(stepCity(jobRepository,txManager))
.next(stepHistorique(jobRepository,txManager))
.build();
}
for the moment, all my @Bean
s are in one class config
@Bean
public Step stepCity(JobRepository jobRepository, PlatformTransactionManager txManager)throws Exception{
return new StepBuilder("stepCity")
.repository(jobRepository)
.<City, City>chunk(25)
.reader(readerCity())
.processor(processorCity())
.writer(writerCity())
.transactionManager(txManager)
.build();
}
@Bean
public Step stepHistorique(JobRepository jobRepository, PlatformTransactionManager txManager)throws Exception{
return new StepBuilder("stepHistorique")
.repository(jobRepository)
.<Historique, Historique>chunk(25)
.reader(readerHistorique())
.processor(processorHistorique())
.writer(writerHistorique())
.transactionManager(txManager)
.build();
}
I did not put what concerns my processor and my other files because it is not important for what I want to do currently
@Bean
public ItemReader<City> readerCity() throws Exception{
FlatFileItemReader<City> readerCity = new FlatFileItemReader<City>();
readerCity.setLineMapper(new DefaultLineMapper() {{
setLineTokenizer(new DelimitedLineTokenizer() {{
setNames(new String[]{"typecom","com","reg","dep","arr","tncc","ncc","nccenr","libelle","can","comparent"});
}});
setFieldSetMapper(new BeanWrapperFieldSetMapper<City>() {{
setTargetType(City.class);
}});
}});
readerCity.setResource(new ClassPathResource("documents/Cities.txt"));
return readerCity;
}
@Bean
public ItemReader readerHistorique() throws Exception{
FlatFileItemReader readerHistorique = new FlatFileItemReader();
readerHistorique.setLineMapper(new DefaultLineMapper() {{
setLineTokenizer(new DelimitedLineTokenizer() {{
setNames(new String[]{"mod","date_eff","typecom_av","com_av","tncc_av","ncc_av","nccenr_av","libelle_av","typecom_ap","com_ap"
,"tncc_ap","ncc_ap","nccenr_ap", "libelle_ap"});
}});
setFieldSetMapper(new BeanWrapperFieldSetMapper() {{
setTargetType(Historique.class);
}});
}});
readerHistorique.setResource(new ClassPathResource("documents/HistoriqueCities.txt"));
System.out.println(readerHistorique);
return readerHistorique;
}
I think it is necessary for everyone to have an ItemReader
@Bean
public ItemWriter<City> writerCity() {
JsonFileItemWriter<City> writerCity = new JsonFileItemWriter<>(
new FileSystemResource("src/main/java/output/Insee.json"),
new JsonObjectMarshaller<City>() {
@Override
public String marshal(City object) {
try {
ObjectMapper mapper = new ObjectMapper();
mapper.configure(SerializationFeature.INDENT_OUTPUT, true);
return mapper.writeValueAsString(object);
} catch (JsonProcessingException e) {
e.printStackTrace();
return null;
}
}
}) {
@Override
public void open(ExecutionContext executionContext)
throws ItemStreamException {
super.open(executionContext);
try {
getOutputState().write(",{Cities: }[");
}
catch(IOException e) {
throw new RuntimeException(e);
}
}
};
writerCity.setResource(new FileSystemResource("src/main/java/output/Insee.json"));
writerCity.setAppendAllowed(true);
writerCity.setJsonObjectMarshaller(new JsonObjectMarshaller<City>() {
@Override
public String marshal(City item) {
Map<String, Object> map = new LinkedHashMap<>();
map.put("status", item.getTypecom());
map.put("insee", item.getCom());
map.put("parent", item.getComparent());
map.put("country", item.getCountry());
map.put("dept", item.getDep());
return map.toString();
}
});
return writerCity;
}
@Bean
public ItemWriter<Historique> writerHistorique() {
JsonFileItemWriter<Historique> writerHistorique = new JsonFileItemWriter<>(
new FileSystemResource("src/main/java/output/Insee.json"),
new JsonObjectMarshaller<Historique>() {
@Override
public String marshal(Historique object) {
try {
ObjectMapper mapper = new ObjectMapper();
mapper.configure(SerializationFeature.INDENT_OUTPUT, true);
return mapper.writeValueAsString(object);
} catch (JsonProcessingException e) {
e.printStackTrace();
return null;
}
}
}) {
@Override
public void open(ExecutionContext executionContext)
throws ItemStreamException {
super.open(executionContext);
try {
getOutputState().write(",{Historique: } [");
}
catch(IOException e) {
throw new RuntimeException(e);
}
}
};
writerHistorique.setResource(new FileSystemResource("src/main/java/output/Insee.json"));
writerHistorique.setAppendAllowed(true);
writerHistorique.setJsonObjectMarshaller(new JsonObjectMarshaller<Historique>() {
@Override
public String marshal(Historique item) {
Map<String, Object> map = new LinkedHashMap<>();
map.put("parent", item.getParent());
map.put("typecom_av", item.getTypecom_av());
map.put("mod", item.getMod());
map.put("nccenr_av", item.getLibelle_av());
map.put("nccenr_av", item.getLibelle_ap());
map.put("date_eff", item.getDate_eff());
return map.toString();
}
});
return writerHistorique;
}
I don’t know if they should each have their own ItemWriter
or if I should write them in just one. City and history each have their class
2
Answers
Yes of course. I have to read my cities.txt file which has this structure:
then I have to read my history.txt file which has this structure:
32,2019-03-01,COM,45287,0,SAINT LOUP DE GONOIS,Saint-Loup-de-Gonois,Saint-Loup-de-Gonois,COMD,45287,0,SAINT LOUP DE GONOIS,Saint-Loup-de-Gonois,Saint-Loup-de-Gonois 32,2019-03-01,COM,45287,0,SAINT LOUP DE GONOIS,Saint-Loup-de-Gonois,Saint-Loup-de-Gonois,COM,45307,3,SELLE SUR LE BIED,Selle-sur-le-Bied,La Selle-sur-le-Bied.
and I have to rewrite them in json format so that each city if it has a "com" number identical to the "com_av" history then following this city its history appears. like that :
I skip the part where I process the data to display only what I need. but for the moment in the code that I gave I read the two files and when I write them I display all the cities and then all the histories. and that's not what I want.
Did you try implement your own ItemReader that fetch both files to the record that you need be written?