From minimal X.org xserver installation on Debian Wheezy, the following is suggested:
apt-get install --no-install-recommends xserver-xorg-core \ xserver-xorg-input-all xserver-xorg-video-fbdev \ xserver-xorg-video-(yourcard, can be intel, nouveau, or ati)
Lately, I have been experimenting with LXDE to get a light weight gui.
Other GUI options are described at Debian DesktopEnvironment
2018/03/28 - Installing a light weight lxde interface into a virtualbox container with a minimal install incorporating the kernel from Stretch backports:
apt install --no-install-recommends \ xserver-xorg-core xserver-xorg-input-all xserver-xorg-video-fbdev \ lxde virtualbox-guest-x11 xserver-xorg-video-vesa \ linux-image-4.14.0-0.bpo.3-amd64