During the process of generating server keys, client keys, or PGP keys in the JSCAPE MFT Server Manager, you're asked to choose between two supported key algorithms: RSA or DSA. Which one should you choose?
The purpose of server, client, and PGP keys
Maybe I should start by explaining very briefly what these keys are for. When you generate a server, client, or PGP "key", you are actually generating a pair of keys: a public key and a private key.
Private keys are for decrypting and signing, while public keys are for encrypting and verifying.
When you generate a server key, the resulting public/private key pair will be used for encryption and decryption purposes during secure file transfers like FTPS, SFTP, and HTTPS. The public key will be used to encrypt the files at the client side and the private key will be used to decrypt those files when they arrive at the server.
On the other hand, the resulting public/private key pair of a client key generation will be used for signing and verification (a.k.a. authentication) purposes at the start of secure file transfers. For example, at the start of an SFTP session, the user can use his private key to "sign" at the client side. The managed file transfer server will then use the corresponding public key stored at the server side to verify the digital signature and, consequently, the user's identity.
The public/private key pair that results from a PGP key generation is more versatile, for it is used for both encryption & decryption as well as in signing & verification.
If you want a more detailed discussion on server and client keys, I encourage you to read the article "Roles of Server and Client Keys in Secure File Transfers".
What DSA and RSA are best suited for
Now it's time to talk about the DSA and RSA algorithms.
Before I proceed, I'd like to point out that, given equal key lengths, the cryptographic strengths of these two algorithms are just about the same. So if the strength of your encryption is all that matters to you, then there's no need to read the rest of this article. You can pick a key algorithm randomly if you want.
However, if performance is an issue (maybe because you're using old machines or you simply have thousands of users simultaneously doing secure file transfers) and there's really a need to eke out computing resources, then read on.
Although their acronyms make it appear that they are very much related, the SA in DSA and the SA in RSA actually don't mean the same thing. RSA stands for Ron Rivest, Adi Shamir, and Leonard Adleman, the people who designed this encryption algorithm. DSA, on the other hand, stands for Digital Signature Algorithm.
The meaning of DSA should give you an idea what it is for (or to be more accurate, what it was originally designed for). As its name implies, DSA was originally intended only for signing. But now, it can be used for encrypting as well. RSA, on the other hand, has long been used for both encryption and signing.
Here are some performance-related facts about RSA and DSA:
Faster at generating keys: DSA
Faster at encrypting: RSA
Faster at decrypting: DSA
Faster at generating a digital signature or "signing": DSA
Faster at verifying a digital signature: RSA
Note that "faster" also means "less demanding on computational resources".
Now, let me try and put in my 2-cents worth based on those facts.
Let's start with key generation. It's generally faster to generate DSA keys. But then, key generation is usually only done once for server keys and once per user for client keys. So I don't really count this as a big point against RSA even if we only want to make comparisons based on performance.
Which one to use for Server Keys
Server keys are used for encryption/decryption. Encryption is done on the client, while decryption is done on the managed file transfer server. But then RSA encrypts faster, while DSA decrypts faster. So how do we go about with this?
Well, if the machines on which your file transfer clients run on are mostly slow, while your server is a powerful one, then it's on the client side where you'll need to eke out computational resources. So in this particular case, you would need server keys that have smaller computational requirements for encryption. Thus, you would need to use RSA.
However, if it's your server that's slow, then you would need server keys that have smaller computational requirements for decryption. Meaning, you will want to use DSA.
Note however that the public/private keys are only used to encrypt/decrypt the session key at the start of a secure file transfer session. It is the session key (a symmetric key) that is actually used to encrypt/decrypt files as they are transmitted between the server and client during the entire duration of the file transfer session. Thus, the difference in encryption/decryption speeds between RSA and DSA will only have an impact at the start of a session.
Which one to use for Client Keys
Again, remember that client keys are used for authentication (e.g. public key authentication). Signing is done on the client side and verification is done on the server side. Now, since DSA is faster at signing and RSA is faster at verifying, we can use the same arguments that we used for server keys.
If your performance issues are more pronounced at the client side, then use DSA. But if your performance issues are on the server side, then use RSA. Another thing to consider is that DSA-generated signatures are significantly smaller. Thus, if you've got limited bandwidth, you might want to use DSA.
Like encryption and decryption, signing and verification are only done at the start of a session.
Which one to use for PGP Keys
In JSCAPE MFT Server, PGP keys are often used with triggers and just stored on the server. As a result, all of the processing (whether for encryption/decryption or signing/verification) is done on the server side. Since RSA and DSA each have their own strengths in this scenario, it's practically a toss-up between the two.
Although DSA and RSA have practically the same cryptographic strengths, each have their own advantages when it comes to performance. DSA is faster at decrypting and signing, while RSA is faster at encrypting and verifying. So if you often encounter performance issues, it might be a good idea to look at where the problem lies (i.e., whether it's client-based or server-based) and base your choice of key algorithm on that.
Still haven't tried JSCAPE MFT Server?
JSCAPE MFT Server comes with a FREE, fully-functional evaluation edition which you can download right now.