Finally after updating to Mavericks I thought I’d have another crack at changing the resolution of my headless Mac mini without using the VGA adapter hack. It is now much easier, as long as you know the trick to it.
First off you need an app to change the screen resolution (scrutil no longer worked for me so I downloaded Display Menu (free) from the Mac App Store).
After changing the display resolution my VNC/Screen Sharing sessions kept blacking out and I couldn’t see anything so I fired up SSH and killed the screen sharing agent (killall ScreensharingAgent) and bam finally my remote screen lit up at my specified resolution (2560×1440). However my Dock was still in the middle of the screen (and killing the Dock seems to reset the resolution), so just right clicking the dock and changing its location made it fix itself up 🙂
Source: Mac OSX Hints