There are several possible reasons why you might want to copy files from Azure Blob Storage to AWS S3 (or vise versa). Maybe you want to migrate from one cloud to another. Maybe you subscribe to both cloud storage solutions and want to synchronize between the two. Maybe you just want to perform a backup. Whatever reason you might have, here's a solution that doesn't require a single line of code, can run fully automated, and can be used for other cloud storage and network services as well.
In this tutorial, we're going to show you how to configure JSCAPE MFT Server so it can copy files from Azure Blob Storage to AWS S3. We've put together several tutorials like this, so just click that link if you want to see more.
In the meantime, let's focus on what we originally set out to do.
Watch the video
Would you prefer to watch a video version of how to download files from the Azure Blob Storage? You can play the video below. Otherwise, just skip it if you wish to continue reading.
First, let me show you what I have on the Azure Blob Storage side. I've launched Microsoft Azure Storage Explorer and navigated to the Azure Blob container named 'jcpv-test' and into the folder named 'folder1'. This is the folder whose contents I want to copy to AWS S3.
Speaking of which, this is my AWS S3 bucket named 'jscapejohn' and in it is a folder also named 'folder1'. Now, the fact that they both have the same names is just coincidental. The source and destination folders don't have to have the same name.
Now that you've seen my Azure Blob storage container folder and my AWS S3 bucket folder, let's now head over to my JSCAPE MFT Server instance and start setting things up there.
I've already logged in to my JSCAPE MFT Server administrative web user interface and navigated to the Trading Partners module in one of my domains.
Here, I have already created trading partner objects for each of the cloud storage solutions involved in this tutorial. I have a trading partner for Microsoft Azure Blob Service and a trading partner for Amazon S3. The former is named 'tp-azureblob' and the latter is named 'tp-s3'.
If you want to know how create these trading partners, you may use these posts for reference:
How to Mirror a FTP Server to S3 Storage
How To Upload Files To Azure Blob Storage Via FTP/S
Not only does JSCAPE MFT Server support a wide range of cloud storage solutions, it also supports several file transfer and network services. This means, you will rarely have any problem transferring files from any source to any destination. As of this writing, the supported cloud storage solutions and network services include:
- AFTP
- AS2
- Amazon S3
- Amazon AWS
- Box
- Dropbox
- Google Cloud
- Google Cloud Storage
- Google Drive
- IBM Cloud
- Local Directory
- Microsoft Azure File Service
- Microsoft Azure Blob Service
- Microsoft Azure Data Lake
- FTP
- FTPS implicit
- FPTS explicit
- IMAP
- OFTP
- POP3
- JSCAPE REST
- SFTP/SCP
- SMB
- SMTP
- WebDAV
- TFTP
Once you have your trading partners ready, the next step is to create a trigger that would be responsible for copying files from your Azure Blob trading partner to your AWS S3 trading partner.
To do that, go to the Triggers module and click the Add button.
For those who are using JSCAPE MFT Server version 12.1 and higher, you'll be presented with a Trigger Template dialog where you can choose a template that best describes the workflow you're about to automate. Let's just leave that blank and click OK.
Give the trigger a name, for example 'copy azure blob to aws s3', and then select the Current Time event type. You may select any event type that suits your needs but, for this example, I'd like this trigger to run at a certain time of the day. That's why I'm using the Current Time event type.
Click Next to proceed.
In the next screen, you can use the Expression Builder to build an expression that would define the schedule when this particular trigger should fire. I want this trigger to execute at 2:35 AM everyday, so I built the expression as shown on the screen.
Recommended read: Introducing the New Trigger Conditions Expression Builder
Click Next to proceed.
Once you get to the next screen, click the Add button to add a new trigger action and then select Trading Partner Synchronization from the Action drop-down list and then click OK.
Let me now walk you through the key settings for this Trading Partner Synchronization action.
The first ones you'll encounter are PartnerA and PartnerB. PartnerA is basically the source. It's the trading partner from which the target trading partner will be copying from. So, in our case, that would be the Azure Blob trading partner or 'tp-azureblob'.
PartnerB, on the other hand, is the target. It's the trading partner to which the source trading partner will be copying to. In our case, that would be the AWS S3 trading partner or 'tp-as3'.
Next up are PathA and PathB. PathA is the relative directory path in PartnerA which PathB will be copying from, and PathB is the relative path on PartnerB that PathA will be copying to.
So, PathA is 'jcpv-test/folder1' and PathB is 'jscapejohn/folder1'. You can review these paths in the screenshots for the two cloud storage solutions we showed you earlier
The next setting we need to specify is the Copy Condition. This is the condition JSCAPE MFT Server will use to determine whether to commence copying (or synchronizing) files each time the predefined schedule of this trigger is up.
If you select:
- different time, JSCAPE MFT Server will commence copying if it sees that file timestamps on A are different from the ones on B;
- different size, JSCAPE MFT Server will commence copying if it sees that file sizes on A are different from the ones on B;
- different content, JSCAPE MFT Server will commence copying if it sees that the content in A is different from the content in B
Let's just choose different time for now.
Another setting you need to specify is the Synchronization Mode. There are four options:
- mirror - New and modified files from A are copied to B; redundant files in B will be deleted;
- synchronize - New and modified files from both paths are copied to each other
- backup - All files from A are copied to B
- contribute - New and modified files from A are copied to B
Based on the descriptions given and your particular use case, you'll likely have to choose between backup and contribute. I'm going to choose contribute, so my trigger is only going to transfer files from A that are new new and/or modified.
Lastly, you need to specify the Result Directory. This is where the results of the copying process will be written to.
Click OK and then drag an arrow from the Start output of the Workflow node to the Trading Partner Synchronization Action node.
Recommended read: Introducing the Redesigned Trigger Action Workflow
Click OK to finalize the trigger creation process.
That's it. Now you know how to configure JSCAPE MFT Server so that you can copy data from Azure Blob Storage to AWS S3.
Try this yourself
Would you like to try this yourself? Download the FREE, fully-functional Starter Edition of JSCAPE MFT Servernow.
Download JSCAPE MFT Server Trial