Do you absolutely need graphical access to your remote server?
I manage my remote Debian servers with
ssh. Most linux servers will be running an ssh daemon waiting for connections, and most linux distributions will have an ssh client installed by default, so you wouldn't need to install anything to run it.
In the linux (and free unix) world, you'll most often come across the
OpenSSH implementation.