Whole grains are way better than refined grains, because they have SOME nutritional benefits-- vitamins and fiber. I think a little bit of whole grains can be beneficial, the same way a little bit of orange juice can be, as long as the proportions are reasonable. A sprinkle of toasted wheat germ or oat bran is tasty mixed with ground flax cereal, providing some good fiber and minerals for a reasonable number of carbs.

Of course, that's way extra-crunchy compared to the "whole grain" breads and pastas, and probably not what you were eating before!