How about: "No, thanks, I'm allergic to sugar/starchy foods...they make me break out in fat."

Seriously...do we really need a label to give legitimacy to refusing a food that is being offered to us?
Yesterday I turned down an ice cream bar with "No, thanks, I don't eat sugar" and left it at that.