To remove just xfce4 package itself from Ubuntu 16.04 (Xenial Xerus) execute on terminal:
sudo apt-get remove xfce4
Uninstall xfce4 and it's dependent packages
To remove the xfce4 package and any other dependant package which are no longer needed from Ubuntu Xenial.
sudo apt-get remove --auto-remove xfce4
If you also want to delete configuration and/or data files of xfce4 from Ubuntu Xenial then this will work:
sudo apt-get purge xfce4
To delete configuration and/or data files of xfce4 and it's dependencies from Ubuntu Xenial then execute:
sudo apt-get purge --auto-remove xfce4
More information about apt-get remove
Advanced Package Tool, or APT, is a free software user interface that works with core libraries to handle the installation and removal of software on Debian, Ubuntu and other Linux distributions. APT simplifies the process of managing software on Unix-like computer systems by automating the retrieval, configuration and installation of software packages, either from precompiled files or by compiling source code.
apt-get is the command-line tool for handling packages, and may be considered the user's "back-end" to other tools using the APT library.
apt-get remove is identical to install except that packages are removed instead of installed. Note that removing a package leaves its configuration files on the system. If a plus sign is appended to the package name (with no intervening space), the identified package will be installed instead of removed.