I had the opposite problem, probably because I gained weight relatively fast. I kept getting sizes that were waaaaaay too small. I would take a 14 off the rack when I fit a 20. When the 14 didn't I could barely hold back the tears.

When i was slimmer, I also tended to think I would wear a smaller size. The only store that has sizes that match my perception is the Gap, which is why I love shopping there