You want to know something that will take over your garden if you're not vigilent...try raspberries! I started a small patch of about 15 canes 4 years ago and those suckers are spreading everywhere! I may have to pull them out if I can't get them contained this year. I also have a bed of asparagus that I get to harvest twice a year, once in the spring and again a smaller harvest in the fall.
One thing that everyone seems to be able to grow that will just not in my garden is zucchini; they get to a certain point and just wither and die. It's okay, though...in the summer around here enough people are trying to get rid of their excess to keep me well supplied.
I've never tried rosemary, but I have a couple of very nice sage bushes that come back every year.