Blog

Managed File Transfer and Network Solutions

How to Monitor SFTP Directories & Download Files

Posted by John Carl Villanueva on Thu, Aug 20, 2020 @ 12:04 AM

In this tutorial, we're going to show you how to monitor directories on a remote SFTP server and then download any new files on a periodic basis. 

[This is on the remote SFTP server]

So now, I'm here on my remote SFTP server where I have a folder named 'folder1' and 3 subfolders underneath it. What I want is to monitor all three folders for any newly added files and then download those files to my JSCAPE MFT Server instance.

 

remote subdirectories in sftp server

 

Watch the video 

Would you prefer to watch a video version of this SFTP download directory tutorial instead? You can play the video below. Otherwise, just skip it if you wish to continue reading.

 

 

Let me take note of this remote SFTP server's IP address, as I'll be needing this when I set up my SFTP trading partner object.  

 

ip address of remote sftp server

 

Ok. Now that we have all that information, let's head to our JSCAPE MFT Server instance so we can set things up there. 

[This is on my JSCAPE MFT Server instance]

Here in my JSCAPE MFT Server instance, I'm going to set up an SFTP trading partner object that points to that remote SFTP server. We already have countless articles showing how to create a trading partner object, so let me just share some links in case you don't know how to create one.

That last article might seem pretty similar to this article but it's not. In that older article, files are downloaded from a remote directory regardless whether they are new files or not. In this new article, MFT Server's will only be downloading new files.

 

sftp trading partner-1

 

Once you have your SFTP trading partner ready, you can then proceed to create a directory monitor that will monitor those remote folders in that trading partner for newly added files. 

Go to the Directory Monitors module and click Add to add a new directory monitor.

 

add directory monitor

 

Next, specify some directory monitor parameters. Start by giving this directory monitor a name, e.g. 'dirmon1'.

Next, tick the Use trading partner checkbox and then select the trading partner you created earlier from the adjacent drop down list. 

In the Directory field, specify the parent directory of those remote folders you want to monitor. In my case, that would be '/folder1'. 

Note: Don't forget the forward slash before the folder name (e.g. /folder1). I forgot that in my screenshot.

Next, tick the Monitor recursively checkbox. This will tell JSCAPE MFT Server to also monitor all subdirectories underneath the folder you specified in the Directory field.

After that, untick the Monitor interval checkbox. We're doing this because we want to control the time this directory monitor fires using a trigger instead of this monitor interval setting. As you will see shortly, doing so will give us more flexibility in specifying a schedule for when this directory monitor should execute. 

 

add directory monitor parameters

 

Lastly, since we just want to monitor these folders for newly added files, let's just leave the Monitor file add checkbox checked and uncheck all other settings under the Events panel.

 

directory monitor file add setting

 

When you're done, click OK until you're back at the main screen.


newly added directory monitor

 

As mentioned earlier, we want to control the execution of that directory monitor using a trigger. Let's create that trigger now. Go to the Triggers module and click the Add button.

 

add new trigger-1

 

If you're using v12.1 and higher, you'll be shown a Trigger Template dialog. Just leave the template untouched and click OK to proceed.

 

trigger template new trigger

 

In the trigger parameters dialog, start by giving this trigger a name, e.g. 'dirmon1activator'.

Next, select the Current Time event type and click Next.

 

add trigger directory monitor activator

 

Here, we can now specify the schedule at which our directory monitor should fire. You can specify specific times of day, days of the week, months, and so on. In my case, I want this trigger to fire (and the directory monitor to fire after it) on the first minute of every hour. 

To do that, I just configure the Expression Builder as shown on the screenshot below. This will generate the expression:

Minute = 1 

Click Next to proceed.

 

new trigger expression fire every hour

 

When you get to the Trigger Actions dialog, click Add to add a new trigger action.

 

add trigger action-1

 

What we want is the Run Directory Monitor Action, so that when this trigger executes, it will in turn run the directory monitor we specify in the next screen. Click OK. 

 

run directory monitor action

 

Once you get to the trigger action parameters dialog, all you need to do is select the directory monitor you created earlier from the Directory Monitor Name drop down list. 

When done, click OK.

 

run directory monitor action parameters

 

Extend an arrow from the Workflow node to the trigger action you just created and then click the OK button to finalize the trigger creation process. 

 

new run directory monitor action

 

You should then see your newly created trigger in your list of triggers.

 

newly added directory monitor trigger

 

Note that what we've created is just the trigger that would fire our directory monitor. We still need to create a trigger that would in turn download new files whenever any new files are detected by the directory monitor during its execution. 

So, again, we click Add to add the second trigger. 

 

add another trigger

 

Give this second trigger a name and then select Directory Monitor File Added from the Event type drop-down list. Click Next to proceed.

 

trigger that responds to directory monitor file added event

 

When you get to the trigger conditions dialog, specify some trigger conditions that would make this trigger respond only to the specific events you want it to respond to. In this example, I'm just going to specify the MonitorName like so. 

MonitorName = "dirmon1"

This will ensure that this trigger will only respond to a directory monitor file added event that originated from the dirmon1 directory monitor. 

Click Next to proceed.

 

trigger conditions for directory monitor file added trigger

 

Again, click Add to add a new trigger action.

 

add new trigger action 2

 

But this time, select the Trading Partner File Download Action

 

add trading partner file download action

 

When you get to the action parameters dialog, select the same trading partner you used for your directory monitor. In my case, that would be tp-sftp. 

Next, enter the %File% variable into the Remote File field. 

And then enter the desired local destination path for the downloaded files. I'm just going to append the %Name% variable to one of my local directories likeso:

C:\downloadsfromsftp\%Name%

Ok. Let me explain what's about to happen here. 

 

trading partner file download action parameters

 

Each time the predefined schedule for our directory monitor execution is up, our directory monitor will fire. When it fires, it will go to those monitored folders (subfolder1, subfolder2, and subfolder3) and check whether any new files were added. Actually, it will also check folder1, so you should bear that in mind. 

If any new files are found, a corresponding Directory Monitor File Added event will be raised for each newly added file. So, if 100 files were added, then the Directory Monitor File Added event will be raised 100 times and the trigger that's listening to this event (in our case, the 'dirmon1activator' trigger) will be fired 100 times as well.

Now, each of these events will have its own set of properties. Two of these properties are the file path and filename of each newly added file. These properties will be stored in the %File% variable and the %Name% variable respectively. So, again, if there were 100 newly added files, then there will be 100 different values for the %File% and %Name% variables. 

So, in turn, the values of the Remote File and Local File fields will vary depending on the values of those variables during a particular trigger execution. 

I hope all that makes sense. Anyway, when you're done click OK to proceed. And OK again until you're back at the main screen.

 

newly created trading partner file download trigger action

 

 

newly created directory monitor file added trigger

 

That's it. Now you know how to configure JSCAPE MFT Server to monitor remote SFTP directories and then download any new files on a periodic basis. 

Get started

Want to try this out? Download the free, fully-functional Starter Edition of JSCAPE MFT Server now.

Download Now

 

 

Topics: JSCAPE MFT Server, Managed File Transfer, Tutorials, SFTP