It's been some time since the last time i used scp to download files so i'm going to take advantage to write down what i need to do. I've got my ssh key setup to be able to connect to the server without password (won't describe how to do that here) and my config is setup knows how to associate my ssh key with the server so:

scp ~/

And that's it! next time i forget, i'll know where to look for when it comes to downloading files with scp.