Step by step:
1. Shut down your VM and edit it's video card settings to use at least 8MB (32MB recommended) of total video RAM.
2. Boot up VM login to the machine and open Device Manager. Expand Display Adapters and right click your video adapter (should be "VMware SVGA II") select "Update Driver Software".
3. Choose "Browse my computer for driver software" and navigate to C:\Program files\Common files\VMware\drivers\wdda_video. Complete the wizard and restart VM when promted.
If you can not find wdda_video folder in your VMware\drivers folder that probably means you are using VMware tools version below 4.1 if that is the case you can download 4.1 tools from here: http://packages.vmware.com/tools/esx/4.1latest/windows/x64/index.html
install it on some test machine and then share "wdda_video" folder to access it from 2008 R2 machine. I don't know what effect it will have running video driver from 4.1 tools package on ESX server below version 4.1 so try it on your own risk.