Yep, that looks like a problem to me. You need them for fiber, to keep your digestive system regular. If not, you could easily be feeling bloated and gain a few pounds just from unprocessed food in your system.

So far all the suggestions made are good ones!