Head to Port Fairy outside of festival time and you'll find a little coastal village with a good beach and very little else. Great if you're after some forced relaxation.
If you're into sustainable energy they also have a riveting wind farm which is open to the public just 20 minutes out of town.
The real reason to visit Port Fairy though is their annual folk festival which has been running for over 30 years, and is so popular that you need to enter a ballot to get a ticket.