I second what Warren said - stop using soap in any form (ie bars, shower gels, bath soaps etc). This will really go a long way in helping your skin.

If you can get one of those soap substitutes like Aqueous Cream, Dermol 500 etc that would be great as these will moisturise your skin otherwise just use water.

Keep using a good moisturiser the rest of the time too.

Eating the way you are is great (unless you're sensitive or allergic to something like dairy but I think you'd probably have figured that out by now).

Did you see a dermatologist? Do they have any idea what it might have been?