The problem with using a single SFTP server
Organizations in different industries like finance, healthcare and retail rely on an SFTP server to transfer files internally and externally. However, using a single SFTP server can result in a failed data transfer workflow, especially if you don’t have a backup plan to correct it. If your organization’s SFTP server experiences hardware issues, glitches, network outages or a security breach, in-progress data transfers may be interrupted, and your data’s integrity could also be compromised. This system issue won’t only lead to communication and security concerns, but it could also cause substantial business disruption because of unexpected and unnecessary downtime. Downtime translates to financial losses, operational delays and reputational damage.
For example, if your organization is in financial services, an inoperable SFTP server could lead to delayed trade confirmations and regulatory penalties. If you work in the healthcare industry, interrupted data exchange could negatively impact patient care. And if your business lies in retail, order processing delays can result in customer churn. This is why it’s important to use a file transfer method, like an HA SFTP solution, that is built with a backup plan in case one of your SFTP servers goes down. Using an HA SFTP solution with built-in redundancy and failover is imperative for ensuring business continuity. If you implement a properly configured load balancer, this can help you manage any data transfer issues by distributing the workload based on your organization’s events.
Day zero ideation: How to assess the risks in advance
When planning for HA, day zero ideation can help you analyze the challenges that are associated with your business using a single SFTP server. Use a risk assessment strategy to identify potential bottlenecks and single points of failure. When analyzing these risks for your organization, ask yourself questions like:
- What happens if our server fails?
- What are the potential points of failure in our current SFTP infrastructure?
- What is the impact of downtime for our company?
- What are our recovery time objectives (RTO)?
- What are our recovery point objectives (RPO)?
It’s also important to consider the volume of your file transfers, peak load periods and other factors you’ll need for scalability. As your business scales, use an SFTP server that helps with load balancing to distribute traffic across multiple nodes. Implementing a load balancer will help manage incoming connections and route them to healthy SFTP server nodes should one fail. You can also explore automation and scripting for server provisioning and configuration to reduce manual effort and human error, while ensuring consistency across nodes.
Take advantage of monitoring features that proactively track server health, performance and potential issues. Document your organization’s entire file transfer architecture, including network configurations, IP addresses and dependencies, so you can pinpoint the different areas and places a transfer might fail. When you implement an HA SFTP server, your business will minimize downtime and thrive even when certain servers fail to transfer important data.
Blocker: User management
User management is one key blocker when it comes to setting up an HA SFTP cluster for your organization. You can use a shared global datastore to synchronize changes and maintain consistency across your organization’s user accounts, permissions and authentication within different nodes. However, you should also consider allowing users to access their home directories and maintain their permissions regardless of which node they use. You can simplify this complex authentication process by using a centralized identity management system with your servers.
Automation and scripting can also help you successfully manage public keys for SFTP users across multiple servers. Proper permissions and file system configurations are also important to ensure consistency across all nodes and prevent errors and data security vulnerabilities. This requires meticulous planning and potentially some command-line work on your Linux or Windows operating systems.
Active-active versus active-passive setups
The choice between active-active and active-passive setups depends on the specific needs of your organization. In an active-active configuration, all nodes actively handle traffic, which allows your organization to better utilize its resources and provides you with faster failover times. In contrast, an active-passive setup uses one main node to handle all the traffic, and there’s at least one more node in standby mode and ready to take over in a failover scenario.
Although an active-passive configuration is simpler to implement, it can prevent your organization from taking advantage of all the resources or nodes available. An active-passive approach can also lead to longer failover times for your business because the standby node will need to be brought online if the initially active node fails.
What we're trying to achieve
Why you would need an SFTP active-active HA cluster
An active-active SFTP HA cluster primarily does two things:
1. It enables load balancing.
and
2. It automatically redirects traffic to one or more healthy server(s) if one of the servers goes down.
Load balancing basically distributes network traffic among all members of the cluster. Although the manner of distribution varies depending on the type of load balancing algorithm used, they all work to prevent a single server from shouldering all of the traffic.
If only one server shoulders the load, that server can easily get overloaded and eventually fail. However, if multiple servers share the load, the chances of one server going down due to overload will be reduced substantially. Ensuring uninterrupted file transfers, even during peak periods, makes this configuration ideal for ”always-on” industries such as finance, healthcare, retail and manufacturing.
An SFTP HA cluster eliminates single points of failure. In the event that a server/node does fail, traffic will be automatically redirected to the remaining nodes in the cluster. The less-busy nodes take on the files so your transfers can continue uninterrupted and without an impact being felt by your clients and customers.
Implement redundancy
One of the basic elements of any HA cluster is redundancy, meaning each node in the cluster must have exactly the same configuration as the other nodes. If all your SFTP services are running on instances of JSCAPE MFT Server by Redwood, then setting up redundancy is simple. Use this tutorial to set up redundancies in your cluster.
First, point each MFT Server (with each running an SFTP service) to a shared global datastore:
To set up a shared global datastore, please see "How to set up an SFTP active-passive cluster." By following the setup directions in that post, all your SFTP servers will automatically acquire the same configurations. Additionally, if you make changes to your settings in one node, those changes will be automatically applied to all other nodes of the cluster.
To balance loads across your cluster, you will need a load balancer. The load balancer will also be responsible for accepting inbound connections and redirecting traffic to available nodes if one of the cluster's nodes fails. We recommend JSCAPE MFT Gateway by Redwood, a software-based load balancer and reverse proxy that supports SFTP as well as several other file transfer protocols that natively work with MFT Server.
Set up the SFTP cluster and apply load balancing
Assuming you already have an instance of JSCAPE MFT Gateway powered up, the first thing you need to do is create an SFTP cluster.
Log in to the JSCAPE MFT Gateway Manager and navigate to the Clusters menu. Click the Add button to add a new cluster.
Give the cluster a name (e.g. sftpcluster) and then select TCP from the Protocol drop-down list.
Choose a load balancing algorithm that suits your needs.
Next, click the Add button immediately below the IP address box to add a host to your SFTP cluster.
Enter the IP address of one of your SFTP nodes as well as its SFTP service's port number (in case it's not the default of 22). Click the Add button.
You should now see the node's connection settings in your list of hosts. Click the Add button again to add another host and repeat for all remaining hosts/nodes.
The image below shows what the screen would look like after adding four nodes.
When you're done adding nodes, click the bottom-most Add button to add the cluster.
To summarize:
- Add an SFTP cluster
- Add nodes to the cluster
- Add and finalize the cluster
Below, we have our newly added SFTP cluster:
The next step would be to add an SFTP service to listen for SFTP connection requests. For example, JSCAPE MFT Gateway would be sitting on your DMZ, with those connection requests coming from an external network, and your SFTP nodes would be sitting inside your internal network:
To add an SFTP service, go to the Services menu and then click the Add button.
Select SFTP/SSH (TCP) for both the Client and Server protocols. This means that clients will be connecting via SFTP, and MFT Gateway will be connecting to remote SFTP services. Click OK to proceed.
Choose an IP address from the local host drop-down list. This would be the IP address of the network interface listening to SFTP client requests. For this example, we chose 0.0.0.0, which stands for all network interfaces on our machine. We also left the port number to the default.
Next, select the Cluster option button and then choose the cluster we created earlier from the drop-down list.
When you're done, click Add or Add/Start. By clicking Add/Start, the reverse proxy service will be immediately enabled as soon as the service is created.
You should then see your newly created SFTP service with a setting indicating its association with the SFTP cluster. This means that all valid SFTP requests connecting to this service will be forwarded to the cluster named "sftpcluster."
Connect all SFTP nodes to a shared storage
Note: This is different from the global datastore.
Lastly, you need to set up a shared storage system for your SFTP cluster. Shared storage allows users and file transfer clients to store and retrieve files to/from the same storage system, regardless of SFTP server connection.
Read more on this in the post "Setting Up a NAS Shared Storage for Your File Transfer Servers". which includes reasons for setup and a detailed example on setting up shared storage.
Get started
Would you like to try this out? Download a free, fully-functional evaluation edition of JSCAPE MFT Gateway now.
Related posts
How To Install A SFTP Server on Windows
Configuring A High Availability Cluster for Various TCP/UDP Services
You Know It’s Time To Implement Server To Server File Transfer When..
Install A Linux SFTP Server Via Command Line and Configure Via Web UI
Introduction to Reverse Proxy Servers
Scheduling Automated File Synchronization Between Two Servers
How To Set Up An Automated SFTP Download
The Quickstart Guide To Setting Up An AS2 Server