Definitely the fat. Fat has more than twice the calories of carbs or protein per gram.

I add 1 tablespoon of butter to my 1/3 cup (measured dry) of oatmeal in the morning, and the butter has as many calories as the oatmeal. Don't skimp on the oil & vinegar dressing on your salads, either. And nuts for snacks -- I think they're practically the perfect food.