Migration of data or files moving around different servers is a common case in software development.
For java and spring boot based solutions, jsch implementations of SFTP are often used. In the following blogpost we will learn about the knowhows in usage and setup of most commonly used java libraries for file uploads and file downloads from remote servers.
SFTP is a secure protocol for file transfers and its fully complaint with the SSH authentication functionalities. This protocol makes authentication on both the server and the user and offers cryptographic hash functions. Appliances of this protocol are file transfers which guarantee data integrity such as migrations of documents or logs, copying data across various servers inside or outside an organization, making automated system backups etc. Everywhere where the file security is integral, this protocol should be used over FTP.
JCraft JSch is a pure java implementation of SFTP protocol which offers easy to setup and easy to use library which covers most of the basic use-cases. It allows connections to remote servers, authentication, port forwarding, secure file transfers, compressing data. Commons VFS is another library which is build on top of JCraft JSch which offers better support, and even more customization. For power users and production enterprise application the latter one is preferred. In the following exercise I will demonstrate, how we can setup a test SFTP server and write java implementation to upload and download files via SFTP.
SFTP vs. FTPS
SFTP runs over the SSH protocol, FTPS is actually the old FTP protocol ran over SSL/TLS. In the end, the FTPS needs much more effort in configuration, when firewall configuration needs to be added and many ports explicitly opened. It also might not work in NAT context. SFTP on the other hand, does not require additional configuration on the server, SSH standard port 22 is the port in which this communication is achieved. For SFTP certificates are not centralized, meaning no additional support and maintenance is expected for them, they can be utilized by any method which use SSH and by any certificate authority. SFTP can be used as a filesystem and various configuration can be added to it, extending use cases. Before we dig into details in the java setup, lets create a docker compose file.
Docker
Atmoz/sftp is a easy to use SFTP server, docker image. This docker image can be used as a mock SFTP server for integration tests, or other research and development scenario. User test can login with SFTP and access files in “Documents” folder. If there is a necessity for testing Logging in with SSH keys, then use the volumes to mount publicKey from the host to the users “.ssh/keys/” of the docker container.
version: "2"
services:
sftp:
image: atmoz/sftp
ports:
- "2222:22"
command: test::1001::Documents
volumes:
- "publicKey.pub:/home/test/.ssh/keys/publicKey.pub"
Now that the container is running, lets add some java configuration.
Jsch
- Adding maven dependency
- Setting up JSch
- Upload to SFTP server/Download from SFTP server
So basically there are three important steps. Lets go into details in these steps for Jsch
Step 1. Add maven dependency
<!-- https://mvnrepository.com/artifact/com.jcraft/jsch -->
<dependency>
<groupId>com.jcraft</groupId>
<artifactId>jsch</artifactId>
<version>0.1.55</version>
</dependency>
This comes as a regular entry in the pom.xml file list of dependencies.
We should always try to use latest stable versions. Version 0.1.55 the latest and it covers some basic use cases. Still, if the use case is something more that a proof of concept, lets say some production web service then, there are better alternatives. Different forks of this implementation have much better support and implement OpenSSH. More about them in the later part of this blogpost.
Step 2. Setup JSch
This java library allows public key authentication or password authentication to a remote server. For the following example, lets use the password authentication one.
private ChannelSftp setupJsch() throws JSchException {
JSch jsch = new JSch();
Session jschSession = jsch.getSession(username, remoteHost);
jschSession.setPassword(password);
jschSession.connect();
return (ChannelSftp) jschSession.openChannel("sftp");
}
Step 3. Upload a file to SFTP server/Download file from SFTP server
For uploading local file to a remote server, put() method is available out of the box.
public void uploadFileUsingResources() throws JschException, SftpException {
ChannelSftp channelSftp = setupJsch();
channelSftp.connect();
String localFile = "src/main/resources/sample.txt";
String remoteDir = "Documents/";
channelSftp.put(localFile, remoteDir + "remoteFile.txt");
channelSftp.exit();
}
Or if the source is an InputStream and not a resource file, or we need to bypass setting path to the local file then here is the following snippet. Jsch put() method also can take streams as arguments.
public void uploadFileInputStream() throws JschException, SftpException {
ChannelSftp channelSftp = setupJsch();
channelSftp.connect();
InputStream inputStream = new FileInputStream("c:\\samples\\sample.txt");
String remoteDir = "Documents/";
channelSftp.put(inputStream, remoteDir + "remoteFile.txt");
channelSftp.exit();
}
Apart from supporting uploading file to a remote server, Jsch also offers service method for downloading file from a remote SFTP server. Downloading file is supported by the get() method, which is also very easy to use.
public void downloadFileUsingResources() throws JschException, SftpException {
ChannelSftp channelSftp = setupJsch();
channelSftp.connect();
String remoteFile = "sample.txt";
String localDir = "src/main/resources/";
channelSftp.get(remoteFile, localDir + "localFile.txt");
channelSftp.exit();
}
For enterprise applications, in the context of transferring files between servers, and in the same time making sure that there is no vulnerability for man in the middle attacks, or password sniffing latest encryption functions and algorithms need to be used. Jsch is not enough in this case as the library is not actively maintained. Somewhere in the beginning of this blogpost I mentioned a fork of this library which gets more, and more active users and is build on top of Jsch. The future of Jsch without ssh-rsa is mwiede/jsch. For power users, which need even more customization, alternatives are Commons VFS and SSHJ.
Apache Commons VFS
Apache Commons VFS is a Virtual File System library. It allows uploading of local files to remote servers, it has methods for checks if file exists in remote location. The library supports moving and deleting files from remote host and all of this in secure manner, using SFTP. Commons VFS is basically Jsch but updated, and with cleaner. better API and maintained repository.
Same as the case was in Jsch, the manual for usage describes three steps…
Step 1. Add maven dependency
<!-- https://mvnrepository.com/artifact/org.apache.commons/commons-vfs2 -->
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-vfs2</artifactId>
<version>2.9.0</version>
</dependency>
This comes as a regular entry in the pom.xml file list of dependencies.
Step 2. Setup VSF2
This time, lets use a bit more production like setup. In this configuration class, lets define a file system and lets use a public key authentication. The property privateKey, or the value of it, of course must be encrypted with jasypt for further security, so that we don’t have plain keys exposed in the repository.
@Configuration
@RequiredArgsConstructor
public class SFTPConfiguration {
@Value("${sftp.private-key}")
private String privateKey;
@Bean
public FileSystemOptions remoteFileSystemOptions() throws FileSystemException {
FileSystemOptions fileSystemOptions = new FileSystemOptions();
SftpFileSystemConfigBuilder fileSystemConfigBuilder = SftpFileSystemConfigBuilder.getInstance();
fileSystemConfigBuilder.setUserDirIsRoot(fileSystemOptions, true);
fileSystemConfigBuilder.setPreferredAuthentications(fileSystemOptions, "publickey");
fileSystemConfigBuilder.setIdentityProvider(fileSystemOptions, jSch -> jSch.addIdentity(privateKey, privateKey.getBytes(StandardCharsets.UTF_8), null, null));
return fileSystemOptions;
}
}
Step 3. Upload a file to SFTP server/Download file from SFTP server
FileSystemManager interface is used to create file objects, which are then used as arguments in the copyFrom() method. FileSystemManager is used to locate a FileObject by name from one of those file systems. FileObject is a file, and is used to access the content and structure of the file. The method copyFrom() copies another file, and all its descendants, to the method caller. On another note, the sftpUrl is a construct which consist of the sftp://theUsername@theRemoteHost:22/andThePathToTheFile
public void uploadFileToSFTP() {
FileSystemManager manager = VFS.getManager();
FileObject localFile = manager.resolveFile(
System.getProperty("user.dir") + "/" + localFile);
String sftpUrl = "sftp://" + username + "@" + remoteHost + ":" + remoteHostPort + "/Documents/" + fileName;
FileObject remoteFile = manager.resolveFile(sftpUrl, fileSystemOptions);
remoteFile.copyFrom(localFile, Selectors.SELECT_SELF);
local.close();
remote.close();
}
Or for the other operation of downloading a file from a sftp server:
public void downloadFileFromSFTP() {
FileSystemManager manager = VFS.getManager();
FileObject localFile = manager.resolveFile( System.getProperty("user.dir") + "/" + localDir + "vfsFile.txt");
String sftpUrl = "sftp://" + username + "@" + remoteHost + ":" + remoteHostPort + "/Documents/" + fileName;
FileObject remoteFile = manager.resolveFile(sftpUrl, fileSystemOptions);
localFile.copyFrom(remoteFile, Selectors.SELECT_SELF);
local.close();
remote.close();
}
In this blogpost, we covered two things. We covered how to setup mock SFTP server with docker and also, covered how to use SFTP integration java libraries to securely transfer files from/to remote servers. Write me in the comments sections if you find this useful or if you have further questions.
