Question posted in
The official documentation can be found here.

Redis – What's the procedure to re-download locally deleted files using SFTP Inbound

As per this doc couldn’t find the right process to re-downloading a locally removed file from remote SFTP.

The requirement is, to delete local file which already been fetched from remote SFTP and use sftp-inbound-adapter (DSL configuration) to re-fetch that same file when required. In this implementation, MetadataStore haven’t been persisted into any external system like PropertiesPersistingMetadataStore or Redis Metadata Store. So as per doc, MetadataStore stored in In-Memory.

Couldn’t find any way to remove meta data of that remote file from MetadataStore to re-fetch the locally deleted file using file_name. And don’t have any clue, how should this removeRemoteFileMetadata() callback needs to be implemented (according to this doc).

Configuration class contain followings:

    @Bean
    public IntegrationFlow fileFlow() {
        SftpInboundChannelAdapterSpec spec = Sftp.inboundAdapter(sftpConfig.getSftpSessionFactory())
                .preserveTimestamp(true)
                .patternFilter(Constants.FILE_NAME_CONVENTION)
                .remoteDirectory(sftpConfig.getSourceLocation())
                .autoCreateLocalDirectory(true)
                .deleteRemoteFiles(false)
                .localDirectory(new File(sftpConfig.getDestinationLocation()));

        return IntegrationFlows
                .from(spec, e -> e.id("sftpInboundAdapter").autoStartup(false)
                        .poller(Pollers.fixedDelay(5000).get()))
                .channel(MessageChannels.direct().get())
                .handle(message -> {
                    log.info("Fetching File : " + message.getHeaders().get("file_name").toString());
                })
                .get();
    }
Tags:

2

Answers


  1. 0

    Use a ChainFileListFilter, with a SftpSimplePatternFileListFilter and a SftpPersistentAcceptOnceFileListFilter.

    Use a SimpleMetadataStore to store the state in memory (or some other MetadataStore).

    new SftpPersistentAcceptOnceFileListFilter(store, "somePrefix");

    Then, store.remove(key) where key is somePrefix + fileName.

    Use a similar filter in the localFilter with FileSystemPersistentAcceptOnceFileListFilter.

    Login or Signup to reply.
  2. 0

    I tried to solve this and I used Tanvir Hossain‘s reference code. I coded like this. 

    @Bean
public IntegrationFlow fileFlow() {
    SftpInboundChannelAdapterSpec spec = Sftp
            .inboundAdapter(sftpConfig.getSftpSessionFactory())
            .preserveTimestamp(true)
            .filter(sftpFileListFilter())
            .localFilter(systemFileListFilter())
            .remoteDirectory(sftpConfig.getSourceLocation())
            .autoCreateLocalDirectory(true)
            .deleteRemoteFiles(false)
            .localDirectory(new File(sftpConfig.getDestinationLocation()));

    return IntegrationFlows
            .from(spec, e -> e.id("sftpInboundAdapter").autoStartup(false)
                    .poller(Pollers.fixedDelay(5000).get()))
            .channel(MessageChannels.direct().get())
            .handle(message -> {
                log.info("Fetching File : " 
                        + message.getHeaders().get("file_name").toString());
            })
            .get();
}


private FileSystemPersistentAcceptOnceFileListFilter systemFileListFilter() {

    return new FileSystemPersistentAcceptOnceFileListFilter(store(), prefix);
}


private ChainFileListFilter<ChannelSftp.LsEntry> sftpFileListFilter() {

    ChainFileListFilter<ChannelSftp.LsEntry> chainFileListFilter = 
                                                new ChainFileListFilter<>();
    chainFileListFilter.addFilters(
            new SftpPersistentAcceptOnceFileListFilter(store(), prefix),
            new SftpSimplePatternFileListFilter(sftpConfig.getFileFilterValue())
    );
    return chainFileListFilter;
}

@Bean
public SimpleMetadataStore store() {
    return new SimpleMetadataStore();
}

    and my Controller for removing metadata is like below :

    public class Controller { 

    private final SimpleMetadataStore simpleMetadataStore;

    public Controller(SimpleMetadataStore simpleMetadataStore) {

           this.simpleMetadataStore = simpleMetadataStore;

    }

    @GetMapping("/test/remove-metadata/{type}/{fileName}")
    @ResponseBody
    public String removeFileMetadata(
             @PathVariable("fileName") String fileName,
             @PathVariable("type") String type
    ) {
        String prefix = definedPrefix;
        String filePath = "";
        if(type.equals("local")){
            filePath = "/local/storage/path/" + fileName;
        }else if(type.equals("remote")){
            filePath = fileName
        }
        String key = prefix + filePath;

        simpleMetadataStore.remove(key);
        return key;
    }

}

    I am getting my desired file. It is re-fetching file for me.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search