Blog

Managed File Transfer and Network Solutions

How To Copy Data From Azure To S3

Posted by John Carl Villanueva on Wed, Sep 16, 2020 @ 06:51 AM

If your business stores data in multiple cloud storage solutions, chances are you've already encountered many instances when you had to transfer data from one cloud solution to another. In this post, you'll learn an easy way to do that using JSCAPE MFT Server. As an example, we'll show you how to set up JSCAPE MFT Server so that you can automatically copy data directly from Azure File Service to Amazon S3. 

Note: Although JSCAPE MFT Server will be orchestrating the copying process from Azure to S3, none of the files being copied will be stored on your JSCAPE MFT Server host. All files involved will be copied directly from Azure to S3.

Watch the video 

Would you prefer to watch a video version of this tutorial instead? You can play the video below. Otherwise, just skip it if you wish to continue reading.

 

 

One advantage of using JSCAPE MFT Server is that, because you already use it for a wide range of file transfer workflows, adding cloud-to-cloud data transfers into the mix further simplifies management of your organization's file transfer processes. 

Recommended read: How to Transfer Files Between Cloud Storage Services

Alright, let's get this tutorial on the road. I'm going to assume you already have an Azure File Service account and an Amazon S3 account. 

If you already have those, login to your JSCAPE MFT Server administrative web user interface and go to the Trading Partners module. Click Add to add a new trading partner.

 

copy data from azure to s3 - add trading partner

 

We need two trading partners here. An Azure File Service trading partner and an Amazon S3 trading partner. We already showed you how to set up an Amazon S3 trading partner in the post  How To Mirror an FTP/S folder to S3, so I suggest you read that first if you haven't done so yet.

To add an Azure File Service trading partner, just expand the protocol drop-down list, select Microsoft Azure File Service, and then click OK.

 

copy data from azure to s3 - add trading partner azure file service

 

Give this trading partner a name, say, 'tp-azure'. 

 

copy data from azure to s3 - trading partner azure file service parameters

 

Next, specify your Microsoft Azure Storage account name and its corresponding Access key

 

copy data from azure to s3 - trading partner azure file service parameters 2

 

Click the Test server button to test your connection. If the test succeeded, click OK.

 

copy data from azure to s3 - trading partner azure file service test connection

 

Once you get back to the main screen, you should see your newly created Microsoft Azure File Service trading partner among your list of trading partners. Of course, you should also see your Amazon S3 trading partner. 

 

copy data from azure to s3 - trading partner azure file service newly created

 

Now that you have your Azure and S3 trading partners ready, the next step would be to create the trigger that would copy data from Azure to S3. Go to the Triggers module and click Add.

 

copy data from azure to s3 - trigger add

 

For those who are using JSCAPE MFT Server version 12.1 and higher, you'll be presented with a Trigger Template dialog where you can choose a template that best describes the workflow you're about to automate. Let's just leave that blank and click OK


copy data from azure to s3 - trigger template

Give this trigger a name, e.g. 'copy from azure to s3'. 

After that, specify an event that would initiate this mirroring process. We just want this trigger to execute at a particular schedule, so let's choose the Current Time event type. 

 

copy data from azure to s3 - trigger event type

In the next screen, you can use the Expression Builder to build an expression that would define the schedule when this particular trigger should fire. I want my trigger to execute at 5:59 AM everyday, so I built the expression as shown on the screen. 

Recommended read: Introducing the New Trigger Conditions Expression Builder

 

copy data from azure to s3 - trigger conditions

When you get to the next screen, click the Add button to add a new trigger action. This is the part of the trigger creation process where you would specify how the copying process should be carried out.

Select the Trading Partner Synchronization action from the drop-down list and then click OK.

 

copy data from azure to s3 - trading partner synchronization action

Let me now walk you through the key settings for this Trading Partner Synchronization action. 

The first ones you'll encounter are PartnerA and PartnerB. PartnerA is basically the source. It's the trading partner from which the target trading partner will be copying from. So, in our case, that would be the Azure trading partner. 

PartnerB, on the other hand, is the target. It's the trading partner to which the source trading partner will be copying data to. In our case, that would be the Amazon S3 trading partner. 

Next up are PathA and PathB. PathA is the relative directory path in PartnerA which PathB will be copying from, and PathB is the relative path on PartnerB that PathA will be copying to.

 

copy data from azure to s3 - trading partner synchronization action parameters

 

The next setting we need to specify is the Copy Condition. This is the condition JSCAPE MFT Server will use to determine whether to commence copying  files each time the predefined schedule of this trigger is up.

If you select:

  • different time, JSCAPE MFT Server will commence copying if it sees that file timestamps on A are different from the ones on B;
  • different size, JSCAPE MFT Server will commence copying if it sees that file sizes on A are different from the ones on B;
  • different content, JSCAPE MFT Server will commence copying if it sees that the content in A is different from the content in B

Let's just choose different time for now. 

Another setting you need to specify is the Synchronization Mode. There are four options:

  • mirror - New and modified files from A are copied to B; redundant files in B will be deleted;
  • synchronize - New and modified files from both paths are copied to each other
  • backup - All files from A are copied to B
  • contribute - New and modified files from A are copied to B

Based on the descriptions given and your particular use case, you'll likely have to choose between backup and contribute. I'm going to choose contribute, so my trigger is only going to transfer files from A that are new new and/or modified.

Lastly, you need to specify the Result Directory. This is where the results of the mirroring process will be written to. 

 

copy data from azure to s3 - trading partner synchronization action parameters 2

Click OK and then drag an arrow from the Start output of the Workflow node to the Trading Partner Synchronization Action node. 

Recommended read: Introducing the Redesigned Trigger Action Workflow

Click OK to finalize the trigger creation process. 

 

copy data from azure to s3 - trading partner synchronization action ok

 

copy data from azure to s3 - trading partner synchronization action new

That's it. Now you know how to configure JSCAPE MFT Server so that you can copy data from Azure to S3. 

Try this yourself

Would you like to try this yourself? Download the FREE, fully-functional Starter Edition of JSCAPE MFT Server now. 

Download Now

Related content:

How to Download Newly Added Files from an AWS S3 Folder

Set Up Automated File Transfers with Windows SMB Share

Moving Data Across a Hybrid Cloud Doesn't Have To Cost An Arm and A Leg

How to Sync a Remote FTP Server to S3

 

Topics: JSCAPE MFT Server, Managed File Transfer, Cloud Computing, Tutorials, Triggers