There's a number of resources online on how to change the default language in Visual Studio. However, none of them are correct.
They all state that you should reset your settings to default and then re-setup your settings. Huh?
Instead, if you modify the CurrentSettings.vssettings file, you can keep all your other settings, without the pain of having to re-import everything else.
First, find your settings file, which appears to default to the My Documents\Visual Studio x\Settings\ directory.
Open CurrentSettings.vssettings in a text or XML editor, and search for DefaultProjectLanguage.
You can then change the value accordingly. Here are the two options for VB or C#.
Save the file, and once you open Visual Studio, you're set.
Now isn't that easier?