A tip for anyone using public keys in SSH: you can start up your xorg session using ssh-agent and then have all subsequent connections be authorized by the agent, saving you some hassle of password typing, etc.  Put this in your ~/.xinitrc :

eval `/usr/bin/ssh-agent -c`
(insert line to start up your window manager here)
/bin/kill $SSH_AGENT_PID

(Yoinked from Matthew Dillon on IRC)  Realistically, you should also lock your terminal or otherwise prevent physical access to any workstation where you do this, since it means immediate SSH access to other systems using your identity, for anyone touching that keyboard.

If you’re using Windows, there’s always Pageant.