Actually, 1/2 cup of milk is more in the range of 6 grams of carb plus 1 or 2 grams for the coffee (yes, coffee has carbs) so your lattes could be running you 8 grams of carb each. If you're on induction, that's almost half of your daily carb allowance.
1/2 cup of heavy cream has 3.25 grams of carb as long as there are no added sugars (check the package) and would put your lattes at closer to 5 grams each.
Using dairy (milk OR cream) does seem to cause stalls in some people as well as caffeine, so if you're unhappy with the rate of your weight loss and you've ruled out all other possilble stallers it might be something to consider cutting but you'll have to decide for yourself if giving up your lattes is worth losing a few extra pounds a month (if it really is the lattes slowing you down...a few weeks without should show you whether it is or not).