The problem is that the average person is not only ignorant, they are indoctrinated. They will parrot their indoctrination at you like their evangelism of food, all kinds of ridiculous nonsense. And then you will have to resist the urge to punch them in the head, and it's always annoying to have that feeling for people that you work with or worse, actually love. So usually it's easier just not to mention it at all. Some people choose to put it differently, like, "I avoid sugar, grains and processed foods, because they don't agree with me."