Some people have allergic reactions to wheat/gluten - some it makes them feel bloated - and still some people with recurrent yeast infections have to stay away from it.

Refined grains have an more immediate effect on your blood sugar - they have a higher glycemic index. Here are links to a few good peices of reading material:

Atkins - Whole Grains
Atkins - Glycemic Index