Twitter API Question posted in
The official Twitter API documentation can be found here.

Batch job for spring batch – Twitter-api

I have following job to be processed at certain time interval or on ad-hoc basis.

Steps in the job are:

  1. Call twitter api and gather tweets for bunch of users and write them
    in a file

  2. Read them from a file and Process them

  3. Dump processed
    results in database

I want UI as well, where I can trigger a job on ad-hoc basis, moreover I should be able to provide parameters to it from UI.

Was thinking of Spring batch for this but it is for more of a read -> process -> write kind of job. Here in first step I am generating data which is read by second step. Not sure if I can still use Spring batch for this OR there could be better way for this.

Tags:

2

Answers


  1. 0

    Using spring batch, you can design your job as steps, each step has it’s own reader, processor, writer.

    <job id="yourJobID" >
    <step id="gatherTweet" next="processTweet">
        <tasklet>
            <chunk reader="tweetCollector" writer="tweetFileWriter"/>
        </tasklet>
    </step>
    <step id="processTweet">
        <tasklet>
            <chunk reader="tweetFileWriterReader" processor="tweetProcessor" writer="tweetDataBaseWriter"/>
        </tasklet>
    </step> 
</job>

    For the UI to launch the job you can use spring batch admin.

    Login or Signup to reply.
  2. 0

    Alternatively you could configure your job and steps using Java rather than using XML.

    For example myJob bean is defined below and uses step0, step1 and step2 (the steps are just beans too):

    // Job Definition //
@Bean
public Job myJob(JobBuilderFactory jobs)
{
    return jobs.get("My Twitter Job")
            .incrementer(new RunIdIncrementer())               
            .from(step0()).next(step1()).next(step2())
            .end()
            .build();
}

...

// Custom ItemReader that is Autowired in //    
@Autowired
protected ItemReader<TweetDto> gatherTweetItemReader;

...

@Bean
public Step step0()
{
    return steps.get("Step0 - gatherTweet")
            .tasklet(gatherTweetItemReader)
            .allowStartIfComplete(true) // Always run this step //
            .build();
}

// ... Step1 and Step2 definitions ... //
    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search