Well,
I've yo-yo'd a heck of a lot and am still to some extent. I think it may take a little bit more as we age but it will eventually work.

If clothes are a problem, maybe try a second hand shop. I shop there quite often and it makes me feel a bit better to have clothes the right size even if it is only temporary.