The only reason to do this instead of removing the packages would be because you might want to still sometimes use the box through the GUI.
Disable X.Org
In order to disable the graphical environment, we’ll need to disable GDM, the Gnome Display Manager. In order to do this, you’ll need to run the following command at the terminal:
When you restart your computer, you’ll be presented with a text-mode login prompt instead of the graphical environment.
Run X.Org While Disabled
If you want to run the graphical environment, all you have to do is type the following command from the prompt, making sure to run it as your normal user account.
The annoying gray screen will go away once Gnome is fully started.
Enable X.Org
If you want to re-enable X11 it’s a simple matter of running this command from the terminal:
When you restart, you’ll be presented with the graphical prompt again.