In this tutorial, we're going to show you how to monitor directories on a remote SFTP server and then download any new files on a periodic basis.
[This is on the remote SFTP server]
So now, I'm here on my remote SFTP server where I have a folder named 'folder1' and 3 subfolders underneath it. What I want is to monitor all three folders for any newly added files and then download those files to my JSCAPE MFT Server instance.
Watch the video
Would you prefer to watch a video version of this SFTP download directory tutorial instead? You can play the video below. Otherwise, just skip it if you wish to continue reading.
Let me take note of this remote SFTP server's IP address, as I'll be needing this when I set up my SFTP trading partner object.
Ok. Now that we have all that information, let's head to our JSCAPE MFT Server instance so we can set things up there
[This is on my JSCAPE MFT Server instance]
Here in my JSCAPE MFT Server instance, I'm going to set up an SFTP trading partner object that points to that remote SFTP server. We already have countless articles showing how to create a trading partner object, so let me just share some links in case you don't know how to create one.
- Trading Partner documentation
- Video: Using Trading Partners
- How To Connect and Upload Files To an Amazon S3 Trading Partner
- Downloading Files Periodically From A Trading Partner
That last article might seem pretty similar to this article but it's not. In that older article, files are downloaded from a remote directory regardless whether they are new files or not. In this new article, MFT Server's will only be downloading new files.
Once you have your SFTP trading partner ready, you can then proceed to create a directory monitor that will monitor those remote folders in that trading partner for newly added files.
Go to the Directory Monitors module and click Add to add a new directory monitor.
Next, specify some directory monitor parameters. Start by giving this directory monitor a name, e.g. 'dirmon1'.
Next, tick the Use trading partner checkbox and then select the trading partner you created earlier from the adjacent drop down list.
In the Directory field, specify the parent directory of those remote folders you want to monitor. In my case, that would be '/folder1'.
Note: Don't forget the forward slash before the folder name (e.g. /folder1). I forgot that in my screenshot.
Next, tick the Monitor recursively checkbox. This will tell JSCAPE MFT Server to also monitor all subdirectories underneath the folder you specified in the Directory field.
After that, untick the Monitor interval checkbox. We're doing this because we want to control the time this directory monitor fires using a trigger instead of this monitor interval setting. As you will see shortly, doing so will give us more flexibility in specifying a schedule for when this directory monitor should execute.
Lastly, since we just want to monitor these folders for newly added files, let's just leave the Monitor file add checkbox checked and uncheck all other settings under the Events panel.
When you're done, click OK until you're back at the main screen.
As mentioned earlier, we want to control the execution of that directory monitor using a trigger. Let's create that trigger now. Go to the Triggers module and click the Add button.
If you're using v12.1 and higher, you'll be shown a Trigger Template dialog. Just leave the template untouched and click OK to proceed.
In the trigger parameters dialog, start by giving this trigger a name, e.g. 'dirmon1activator'.
Next, select the Current Time event type and click Next.
Here, we can now specify the schedule at which our directory monitor should fire. You can specify specific times of day, days of the week, months, and so on. In my case, I want this trigger to fire (and the directory monitor to fire after it) on the first minute of every hour.
To do that, I just configure the Expression Builder as shown on the screenshot below. This will generate the expression:
Minute = 1
Click Next to proceed.
When you get to the Trigger Actions dialog, click Add to add a new trigger action.
What we want is the Run Directory Monitor Action, so that when this trigger executes, it will in turn run the directory monitor we specify in the next screen. Click OK.
Once you get to the trigger action parameters dialog, all you need to do is select the directory monitor you created earlier from the Directory Monitor Name drop down list.
When done, click OK.
Extend an arrow from the Workflow node to the trigger action you just created and then click the OK button to finalize the trigger creation process.
You should then see your newly created trigger in your list of triggers.
Note that what we've created is just the trigger that would fire our directory monitor. We still need to create a trigger that would in turn download new files whenever any new files are detected by the directory monitor during its execution.
So, again, we click Add to add the second trigger.
Give this second trigger a name and then select Directory Monitor File Added from the Event type drop-down list. Click Next to proceed.
When you get to the trigger conditions dialog, specify some trigger conditions that would make this trigger respond only to the specific events you want it to respond to. In this example, I'm just going to specify the MonitorName like so.
MonitorName = "dirmon1"
This will ensure that this trigger will only respond to a directory monitor file added event that originated from the dirmon1 directory monitor.
Click Next to proceed.
Again, click Add to add a new trigger action.
But this time, select the Trading Partner File Download Action.
When you get to the action parameters dialog, select the same trading partner you used for your directory monitor. In my case, that would be tp-sftp.
Next, enter the %File% variable into the Remote File field.
And then enter the desired local destination path for the downloaded files. I'm just going to append the %Name% variable to one of my local directories likeso:
C:\downloadsfromsftp\%Name%
Ok. Let me explain what's about to happen here.
Each time the predefined schedule for our directory monitor execution is up, our directory monitor will fire. When it fires, it will go to those monitored folders (subfolder1, subfolder2, and subfolder3) and check whether any new files were added. Actually, it will also check folder1, so you should bear that in mind.
If any new files are found, a corresponding Directory Monitor File Added event will be raised for each newly added file. So, if 100 files were added, then the Directory Monitor File Added event will be raised 100 times and the trigger that's listening to this event (in our case, the 'dirmon1activator' trigger) will be fired 100 times as well.
Now, each of these events will have its own set of properties. Two of these properties are the file path and filename of each newly added file. These properties will be stored in the %File% variable and the %Name% variable respectively. So, again, if there were 100 newly added files, then there will be 100 different values for the %File% and %Name% variables.
So, in turn, the values of the Remote File and Local File fields will vary depending on the values of those variables during a particular trigger execution.
I hope all that makes sense. Anyway, when you're done click OK to proceed. And OK again until you're back at the main screen.
That's it. Now you know how to configure JSCAPE MFT Server to monitor remote SFTP directories and then download any new files on a periodic basis.
Get started
Want to try this out? Download the free, fully-functional Starter Edition of JSCAPE MFT Server now.