Question posted in
Some documentation to get you started can be found here.

Too many connections with apache commons-vfs2

I have an application that needs to download files from sftp.
I’m currently using apache commons-vfs2

I have a scheduler that runs every 1 minute.
1. Get the list of files that’s on remote (open connection, get the list, then close connection)
2. Download the files from step 1 (open connection, download each files, then close connection)

How can I keep the connections to minimum?
Is there a way to limit how many connections I have with commons-vfs2?

Here is my code

private List<FileObject> getRemoteFilesList() throws FileSystemException {
        FileObject[] remoteFiles;
        try {
            manager.init();
            final @Cleanup FileObject remoteDirectoryObject = manager.resolveFile(uri, fileSystemOptions);
            remoteFiles = remoteDirectoryObject.getChildren();

        } finally {
            manager.freeUnusedResources();
            manager.close();
        }
        return Arrays.stream(remoteFiles)
                     .collect(Collectors.toList());
    }

private List<File> downloadRemoteFiles(final List<FileObject> remoteFiles) {
        if(remoteFiles.isEmpty()) {
            return Collections.emptyList();
        }

        final List<File> myCollection = new ArrayList<>();
        try {
            manager.init();

            for (final FileObject myfile : remoteFiles) {
                final File localFile = downloadFile(myfile);
                myCollection.add(localFile);
                myfile.delete();
            }
        } catch (final IOException exception) {
            log.warn("Unable to download because ", exception);
        } finally {
            manager.freeUnusedResources();
            manager.close();
        }
        return myCollection;
    }
Tags:

2

Answers


  1. 0

    The apache commons wiki for VFS (https://wiki.apache.org/commons/VfsFaq) says to use the following when closing an SFTP connection in certain circumstances:

    ((DefaultFileSystemManager) fsManager).close();

    This forces the close method on the DefaultFileSystemManager to be called, rather than the close method on the FileSystemManager class.

    It may not be your issue, but it’s something that may be related.

    Login or Signup to reply.
  2. 0

    You can try this alternative method to clean up any temporary files and close all providers.

    FileObject src = null;

/**
 * Release system resources, close connection to the filesystem. 
 */
public void release() {
    FileSystem fs = null;

    this.src.close(); // Seems to still work even if this line is omitted
    fs = this.src.getFileSystem(); // This works even after the src is closed.
    this.fsManager.closeFileSystem(fs);
}

    More details can be found on: https://cwiki.apache.org/confluence/display/COMMONS/SimpleSftpFileDownload

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search