Well there is the damage to metabolism and body if your weight is yo-yoing. If it's just small cheats and you're weight isn't fluctuating to much then the damage comes in insulin spikes which store the fat your eating, increased inslulin resistance, feeding and strengthening addictions, and the bad emotions associated.

Why not start looking at why you cheat. Are you actually hungry because your not eating enough while "on plan"? Is it a matter of carb addictions? Are you having hidden carbs in your regular diet? Is it social pressures? Where is the food coming from, do you still have junk food and temptations in the house?

Instead of hopign someone tells you it's not to serious of a problem, your just "refeeding", I think it's more important to figure out why you keep going off wagon and if it's just a symptom of something more. Are you sabataging yoruself at a certain weight, or when you recieve compliments or doubters?