Amazon S3 vs Local Storage - Where Should You Store Files Uploaded to Your File Transfer Server?
Learn about the advantages and disadvantages of storing uploaded MFT files using the Amazon S3 storage service as opposed to saving your files locally.
Some modern file transfer servers already have the built-in capability to store uploaded files in different places. For instance, in JSCAPE MFT Server, you have the option to store uploaded files either locally (i.e. on the same machine as the system running the file transfer service itself) or to a remote server or storage service. One of these storage services is Amazon S3. But why would you want to store uploaded files there?
In this post we explore some of the pros and cons of using Amazon S3 as your MFT Server's file storage system compared to the default option, i.e. local storage.
Advantages of using Amazon S3 for file storage
Amazon S3, which stands for Simple Storage Service, is Amazon's cloud-based storage solution. You can connect to it from MFT Server by using MFT Server's reverse proxy module. Because it's a cloud-based solution, S3 comes with the benefits of the cloud, including:
Most file transfer servers store their files in hard disks that are directly attached (or built-in) to the physical server itself. These disks naturally have limited space. That's why server administrators often set storage quotas or adopt regular file deletion or archiving practices in order to conserve disk space.
Once the maximum limit of these local disks is reached, they have to be replaced or additional disks have to be added to the server. This can be time-consuming and can cause a substantial amount of downtime.
By comparison, Amazon's massive and highly scalable infrastructure enable file transfer servers to have virtually unlimited storage. That means, your users and trading partners can upload gigabytes upon gigabytes or terabytes upon terabytes of files and your server won't even bat an eyelash. An Amazon S3 bucket, which is where your uploaded files will be stored, has unlimited capacity. So your users can upload as many files, regardless of size, as they want.
High availability and disaster recovery capabilities
One problem with local storage is that, if the local hard disk crashes, your users will not be able to access their files. Similarly, if the hard disk just works fine but the server itself crashes, the files still won't be accessible either.
You won't be subjected to the same level of risk if you store your files on Amazon S3. That's because, first of all, Amazon S3 is built on top of a reliable, fault-tolerant, and highly available infrastructure. The chances of an Amazon S3 bucket 'crashing' or becoming unavailable is extremely low.
Also, if your server itself crashes, you can still access your files on Amazon S3 by simply using another instance of JSCAPE MFT Server (e.g. running on a backup server) and then connecting to the Amazon S3 bucket using exactly the same S3 login credentials.
If you want, you can even set up two or more instances of JSCAPE MFT Server behind a MFT Gateway reverse proxy to form an active-active high availability clusterand then have those instances connect to the same Amazon S3 bucket to reduce the chances of an MFT Server going down due to overload.
Heavily used file transfer servers that aren't bound by strict storage quota restrictions or file deletion and archiving policies can rapidly gobble up storage resources. That can be a problem if your IT department is running on a tight budget.
Let's say you're anticipating a large storage consumption in the next couple of years. And so, in order to meet that anticipated demand, you purchase an enterprise-class on-premise storage solution. That could easily cost north of a thousand $$$$. And because it's an upfront cost, you'll feel the full force of the expense right away even if you won't be able to make full use of its maximum capacity yet.
An Amazon S3 subscription is more practical and affordable because you only need to pay as you go. Although the prices vary from one region to another, they're usually just a couple of cents per GB.
So if you're starting small, you can certainly take advantage of the substantial cost savings. Then as your storage demands increase, you can just gradually increase your spending. This rate of expense will be easier on your cash flow.
Some data protection laws and regulations impose restrictions to where files should be stored. Some laws, for instance, have data localization provisions that prohibit files from being stored overseas or have stringent requirements if files have to be stored outside certain regions (the EU GDPR is one example).
In cases like this, storing files in Amazon S3 provides certain benefits. If the users of your file transfer server are scattered across the globe or located in different regions with varying data protection legislations, you could benefit from Amazon S3's capability to store data in different geographical locations a.k.a. regions.
Basically, when you create an Amazon S3 bucket, you can specify which region that bucket will be hosted in. So, if you have a geographically dispersed user base, then you can create multiple buckets and assign each user to whichever bucket (along with its associated region) keeps you in compliance with the law. Some of the supported regions include:
- US East, in North Virginia
- US West, in Oregon
- EU, in Ireland,
- Asia Pacific, in Tokyo
Managed file transfer servers like JSCAPE MFT Server allow you to connect to multiple buckets and map each bucket (via reverse proxy) to a virtual path. So, you can, for example, enable one set of users to store their files in the US and another set of users to store their files in EU. You can even enable certain users to have access to buckets in different regions. You can't have this flexibility if your server can only store files locally.
This isn't necessarily an advantage over local storage (because you can also make local storage equally secure), but it's worth mentioning nevertheless.
During a file transfer session, i.e., when files are being uploaded to your server, files are protected via data-in-motion encryption technologies like SSL/TLS (in the case of FTPS or HTTPS) or SSH (in the case of SFTP). But once they're already on your server, they will have to be secured with some form of data-at-rest encryption solution.
For locally stored files, JSCAPE MFT Server offers PGP-encryption. But how about files uploaded to an Amazon S3 bucket? For this purpose, Amazon offers server-side encryption. Amazon S3 encrypts data using AES-256, a strong and widely accepted block cipher that's also a federal government standard.
Disadvantages of using Amazon S3 for file storage
Although there are many advantages of using Amazon S3 as the file storage system for your managed file transfer server, it also comes with a few disadvantages. We discuss some of them here.
When it comes to file transfers, latency is always a bad thing. High latency means slower uploads and downloads. Generally speaking, the longer the distance files have to travel, the greater the amount of latency they'll get to experience.
Recommended read: Introducing Accelerated File Transfer (an article that discusses how to increase file transfer speeds even in high latency networks)
Storing files on Amazon S3 means you'll need to deal with additional latency because, after files are uploaded to your server, they'll still have to be forwarded to the Amazon S3 infrastructure. This means, they'll have to go through the Internet again.
That's a much longer path than if the files only had to be stored in a disk directly attached to (or even in the same local area network as) the file transfer server itself. Worse, if your S3 bucket is located in a region far from the geographical location of your server, you'll experience even greater latency.
Of course, if your MFT Server is running on an Amazon EC2 instance, then the latency might be reduced. However, it's probably still going to be higher than if you simply stored files in a (physical) directly attached disk.
In spite the strong security measures implemented by Amazon on the S3 infrastructure, some companies just don't feel comfortable entrusting their sensitive data to third parties. In these situations, you're left with no other option but to store your files locally or in the same data center as the one hosting your file transfer server.
How To Use Amazon S3 As Storage For Your MFT Server
How to Access AWS S3 Storage with SFTP
How To Connect and Upload Files To an Amazon S3 Trading Partner
How to Download Newly Added Files from an AWS S3 Folder
Monitoring an Amazon S3 Folder and Transferring Newly Added Files To Azure
How To Transfer Files Uploaded To An FTP/S Server To An Amazon S3 Bucket
How To Sync a Linux Directory with an Amazon S3 Bucket
How To Copy Files From Azure Blob Storage To AWS S3 | JSCAPE