Overview
An active-active high availability SFTP cluster can greatly reduce the chances of downtime by allowing two or more SFTP servers share the load. If you'd like to learn an easy way of setting up this kind of cluster, check out this post.
Note: This post was originally published on Sept 2016 but was updated and re-published on Oct 2018.
What we're trying to achieve
Why you would need a SFTP active-active HA cluster
An active-active SFTP high availability cluster primarily does two things:
1. It enables load balancing.
and
2. It automatically redirects traffic to 1 or more healthy server(s) if one of the servers go down.
Load balancing basically distributes network traffic among all members of the cluster. Although the manner of distribution varies depending on the type of load balancing algorithm used, they all work to prevent a single server from shouldering all of the traffic.
If only one server shoulders the load, that server can easily get overloaded and eventually fail. This can cause downtime. However, if multiple servers share the load, the chances of one server going down due to overload will be reduced substantially. It also has another benefit.
A SFTP HA cluster eliminates single points of failure. In the event that a server/node does fail, traffic will be automatically redirected to the remaining nodes in the cluster. They can then take up the load and your file transfers can continue as if nothing happened.
Implement redundancy
One of the basic elements of any high availability cluster is redundancy. This means that each node in the cluster must have exactly the same configuration as the other nodes. If all your SFTP services are running on instances of JSCAPE MFT Server, then setting up redundancy is as easy as pie.
All you have to do is point each MFT Server (with each running an SFTP service) to a shared global datastore like so:
We already included a detailed discussion on that in the post "How To Set Up A SFTP Active-Passive Cluster", so I suggest you read that first. If you follow the steps for connecting to a global datastore as instructed in that post, all your SFTP servers will automatically acquire the same configurations. If you make changes to your settings in one node, those changes will be automatically applied to all other nodes of the cluster.
We now proceed to the most crucial part of this tutorial.
Set up the SFTP cluster and apply load balancing
In order to balance loads across your cluster, you need a load balancer. This load balancer will also be responsible for accepting inbound connections as well as redirecting traffic to available nodes should one of the cluster's nodes fail. What we recommend is JSCAPE MFT Gateway, a software-based load balancer and reverse proxy that supports SFTP as well as several other file transfer protocols.
Assuming you already have an instance of MFT Gateway powered up, the first thing you need to do is create a SFTP cluster.
Login to the MFT Gateway Manager and navigate to the Clusters menu. Click the Add button to add a new cluster.
Give the cluster a name (e.g. sftpcluster) and then select TCP from the Protocol drop-down list.
Choose a load balancing algorithm that suits your needs.
Next, click the higher Add button (there are two on that screen) to add a host to your SFTP cluster.
Enter the IP address of one of your SFTP nodes as well as its SFTP service's port number (in case it's not the default - 22). Click the Add button.
You should then see that node's connection settings in your list of hosts. Click the Add button again to add another host. Do this for all remaining hosts/nodes.
Here's how my screen looks like after adding 4 nodes.
When you're done adding nodes, click the lower Add button to add the cluster.
To summarize, first you add a SFTP cluster, then you add nodes to the cluster, and then you add/finalize the cluster.
Here we have our newly added SFTP cluster.
The next step would be to add an SFTP service that would be listening for SFTP connection requests. So, for example, MFT Gateway would be sitting on your DMZ, those connection requests would be coming from an external network, and your SFTP nodes would be sitting inside your internal network.
To add an SFTP service, go to the Services menu and then click the Add button.
Select SFTP/SSH (TCP) for both the Client and Server protocols. This means, clients will be connecting via SFTP and MFT Gateway will be connecting to remote SFTP services. Click OK to proceed.
Choose an IP address from the Local host drop-down list. This would be the IP address of the network interface that would be listing to SFTP client requests. For this example, we just chose 0.0.0.0, which stands for all network interfaces on our machine. We also left the port number to the default.
Next, select the Cluster option button and then choose the cluster we created earlier from the drop-down list.
When you're done, click Add or Add/Start. If you click the latter, the reverse proxy service will be immediately enabled as soon as the service is created.
You should then see your newly created SFTP service with a setting indicating that it's associated with the SFTP Cluster. That means, all valid SFTP requests that connect to this service will be forwarded to the cluster named "sftpcluster".
Connect all SFTP nodes to a shared storage
Note: This is different from the global datastore.
Last but not the least, you need to set up a shared storage system for your cluster. The purpose of having a shared storage is that it allows users and file transfer clients to store and retrieve files to/from the same storage system regardless which SFTP server they manage to connect to.
We already had an extensive discussion on this matter in the post "Setting Up a NAS Shared Storage for Your File Transfer Servers". That post already includes reasons for setting up as well as a detailed example on how to set up a shared storage, so I suggest you take the time to read it.
Get started
Would you like to try this out? Download a free, fully-functional evaluation edition of JSCAPE MFT Gateway now.
Related posts
How To Install A SFTP Server on Windows
Configuring A High Availability Cluster for Various TCP/UDP Services
You Know It’s Time To Implement Server To Server File Transfer When..
Install A Linux SFTP Server Via Command Line and Configure Via Web UI
Introduction to Reverse Proxy Servers
Scheduling Automated File Synchronization Between Two Servers
How To Set Up An Automated SFTP Download
The Quickstart Guide To Setting Up An AS2 Server