Perhaps it's bothering you now because it's as though they are not acknowledging your efforts?
I know you have been working hard, not only on losing the weight, but getting rid of the mental baggage that goes along with it. Perhaps it bothers you particularly now because you want to be seen as a "future slim person" and these comments make it seem like you are not regarded that way.
However, if that's what they are thinking, they are wrong. I don't think you are wrong to feel that way.

The only wrong thing is if you spend any time proving them right.