I think you have the right idea. Eat before you go. There's no law against refusing a dinner invitation. Then meet after dinner.
With family it shouldn't be a problem. You're an adult now and can choose what you eat. As was mentioned above take some dishes to family gatherings so you will have something to choose from. Don't let them pressure you into doing something against your principles.