When using SSL, it will take extra CPU usage and some time to encrypt/decrypt the data. So the 5% CPU mentioned is quite normal and the download/upload speed can be a little slower than unencrypted file transfer. But in your case, the speed difference is too large. I have tested both on "real" OS and the virtual box. The ratio of download speed between http:https on "real" OS is 1:3, and on virtual box is 1:6. The problem also has something to do with the ssl certificate bits (1024, 2048 or 4096). The bigger the bits, the more secure the transfer will be, but this also means more CPU usage and the speed can be really slow.
Try to move your server from Virtual Box to the "real" OS, and test it.
Also, you can test the speed of our demo server using http and https on your side. Then tell us the result.
Please refer to: http://www.wftpserver.com/onlinedemo.htm