As long as you are not experiencing any side effects from the dairy, it should be ok. Especially if you stick to the highest fat dairy. Just be mindful of any joints that are achier (is that a word?), or if your nose gets stuffy, or you chest feels tight, you know all the standard signs of allergies, I'm sure.

I myself had always thought I was lactose intolerant and not sensitive to wheat. But lately I found out this may not be true. I, too, started eating some dairy, and had no problems with it, surprisingly. However, I cheated once with onion rings (batter dipped, of course) recently. By that evening, my chest was so tight I had trouble breathing! So, for me anyway, grain is way worse than dairy. I would much rather eat a little cheese than eating a piece of cake.