Aside from a possible intolerance, dairy can be a problem for some, women in particular, because of two main things - sodium (especially in cheeses) and hormones (especially for women).

Your best bet is usually to try an elimination diet for a few weeks and see how you feel. If it's the culprit you should see a noticeable improvement; you'll also notice symptoms reoccuring if you add it back in.

If the problem is the hormones that are found in cow diary many find that sheep / goat products do not effect them in the same way.

Nat