Blog

Managed File Transfer and Network Solutions

How to Mirror a FTP Server to S3 Storage | JSCAPE

Posted by John Carl Villanueva on Fri, Sep 04, 2020 @ 02:52 AM

If your FTP/S service runs on JSCAPE MFT Server, it's possible to mirror files stored in that FTP server to an Amazon S3 bucket folder. We'll show you how to do that in this post. 

Mirroring in this context is basically a type of synchronization process that ensures a target directory (in this case, the Amazon S3 bucket folder) ends up with exactly the same files as a source directory (in this case, your FTP server).

how to mirror ftp to amazon s3 thumb

Some people don't want to call it synchronization because, strictly speaking, synchronization is a two way process, i.e., new and modified files from both directories are copied onto both paths. Mirroring is normally just one way, i.e., new and modified files are copied from source to target and redundant files in the target are deleted. As you'll see in a short while, JSCAPE MFT Server supports both processes. 

Watch the video 

Would you prefer to watch a video version of this tutorial instead? You can play the video below. Otherwise, just skip it if you wish to continue reading.

 

 

[this is on your Amazon S3 bucket]

I'm going to assume you already have an Amazon S3 bucket with a folder inside it. Here, for example, I have a bucket named 'jscapejohn' and inside it is a folder named 'folder1'. Take note of the region (e.g. US East (Ohio)) indicated for that bucket as you'll be needing that piece of information when you set up your Amazon S3 trading partner on JSCAPE MFT Server. 

 

mirror ftp to s3 - s3 page

 

[this is on your FTP server]

I'm now on my JSCAPE MFT Server instance and, as you can see, I have an FTP service running on it. So, this basically serves as an FTP server. JSCAPE MFT Server supports several other file transfer protocols, so it can also be an SFTP server, an HTTP server, a WebDAV server, and so on, all at the same time.

 

mirror ftp to s3 - ftp service

 

Before we can set this up to mirror a local folder to our Amazon S3 folder, we need to create a couple of trading partner objects first. The first one will be a Local Directory trading partner, which will serve as a sort of reference point for any local directory on this JSCAPE MFT Server instance. Of course, the second one will be our Amazon S3 trading partner.

To create a trading partner, just go to the Trading Partners module and click the Add button.

 

mirror ftp to s3 - add trading partner

 

We'll be creating out Local Directory trading partner first, so just expand the Protocol drop-down list and select Local Directory. Click OK to proceed.

 

mirror ftp to s3 - trading partner local directory

 

Next, give this trading partner a name, e.g. tp-local, and enter any other pertinent information that you wish to specify for this trading partner. We'll just leave ours blank for now. Once you're done click OK. 

 

mirror ftp to s3 - trading partner local directory parameters

 

You can then do the same process for the Amazon S3 trading partner.  

 

mirror ftp to s3 - trading partner amazon s3

 

The Amazon S3 trading partner will have a couple more fields that you'll need to fill up. In addition to the trading partner object's name...

 

mirror ftp to s3 - trading partner amazon s3 parameters 1

 

... you also need to specify your Amazon S3 account's Username or Access Key as well as its corresponding Password or Secret Key.

Lastly, you'll need to specify the region. If you recall, I asked you take note of that earlier. In our case, our Amazon S3 bucket's region is US East (Ohio). 

Click OK to proceed.

 

mirror ftp to s3 - trading partner amazon s3 parameters 2

 

Once you're trading partner objects are ready, you can select them one by one and click the Test button to test if your connections to those trading partners are actually working. 

 

mirror ftp to s3 - trading partners test

 

If all tests succeed, you can then proceed to create a trigger that would enable an automated mirroring process from a directory you will soon specify to your Amazon S3 bucket folder. 

To do that, just go to the Triggers module and click the Add button.

 

mirror ftp to s3 - add trigger

 

For those who are using JSCAPE MFT Server version 12.1 and higher, you'll be presented with a Trigger Template dialog where you can choose a template that best describes the workflow you're about to automate. Let's just leave that blank and click OK

 

mirror ftp to s3 - add trigger template

 

Give this trigger a name, e.g. 'sync local with s3'. 

Now, you can specify an event that would initiate this mirroring process. We just want this trigger to execute at a particular schedule, so let's just choose the Current Time event type. 

 

mirror ftp to s3 - add trigger parameters name event

 

In the next screen, you can use the Expression Builder to build an expression that would define the schedule when this particular trigger should fire. I want this trigger to execute at 11:30 PM everyday, so I built the expression as shown on the screen. 

Recommended read: Introducing the New Trigger Conditions Expression Builder

 

mirror ftp to s3 - add trigger conditions

 

When you get to the next screen, click the Add button to add a new trigger action. This is the part of trigger settings where you would specify how the mirroring process should be carried out.

Select the Trading Partner Synchronization action from the drop-down list and then click OK.

 

mirror ftp to s3 - add trigger action trading partner synchronization

 

Let me now walk you through the key settings for this Trading Partner Synchronization action. 

The first ones you'll encounter are PartnerA and PartnerB. PartnerA is basically the source. It's the trading partner from which the target trading partner will be mirroring from. So, in our case, that would be the Local Directory trading partner. 

PartnerB, on the other hand, is the target. It's the trading partner to which the source trading partner will be mirroring to. In our case, that would be the Amazon S3 trading partner. 

Next up are PathA and PathB. PathA is the absolute local directory path in PartnerA which PathB will be mirroring from, and PathB is the relative path on PartnerB that PathA will be mirroring to.

 

mirror ftp to s3 - add trigger action parameters 1

 

The next setting we need to specify is the Copy Condition. This is the condition JSCAPE MFT Server will use to determine whether to commence copying (or mirroring) files each time the predefined schedule of this trigger is up.

If you select:

  • different time, JSCAPE MFT Server will commence copying if it sees that file timestamps on A are different from the ones on B;
  • different size, JSCAPE MFT Server will commence copying if it sees that file sizes on A are different from the ones on B;
  • different content, JSCAPE MFT Server will commence copying if it sees that the content in A is different from the content in B

Let's just choose different time for now. 

Another setting you need to specify is the Synchronization Mode. There are four options:

  • mirror - New and modified files from A are copied to B; redundant files in B will be deleted;
  • synchronize - New and modified files from both paths are copied to each other
  • backup - All files from A are copied to B
  • contribute - New and modified files from A are copied to B

Obviously, we need to select 'mirror' because that's what we're here for but at least you know there are other options to choose from. 

Lastly, you need to specify the Result Directory. This is where the results of the mirroring process will be written to. 

 

mirror ftp to s3 - add trigger action parameters 2

 

Click OK and then drag an arrow from the Start output of the Workflow node to the Trading Partner Synchronization Action node. 

Recommended read: Introducing the Redesigned Trigger Action Workflow

Click OK to finalize the trigger creation process. 

 

mirror ftp to s3 - add trigger action ok

 

mirror ftp to s3 - newly added trading partner synchronization action trigger

 

That's it. Now you know how to configure JSCAPE MFT Server so that you can mirror an FTP/S folder to an Amazon S3 bucket folder. 

Try this yourself

Would you like to try this yourself? Download the FREE, fully-functional Starter Edition of JSCAPE MFT Server now. 

Download Now

 

Related content:

How to Download Newly Added Files from an AWS S3 Folder

Set Up Automated File Transfers with Windows SMB Share

Moving Data Across a Hybrid Cloud Doesn't Have To Cost An Arm and A Leg

How To Monitor Moved Files Without Using Directory Monitors

Topics: Ad-Hoc File Transfers, JSCAPE MFT Server, Managed File Transfer, Business Process Automation, Cloud Computing, Tutorials, Triggers