I do keep track of them but not fanatically. I can lose (very slowly) at up to 1800 calories per day, but I find that I also do better if I stick closer to 1500. Interestingly enough, yesterday I was over 1600 and I had a nice 2 pound whoosh overnight.
Mostly, I have to keep track to make sure I'm eating enough. Like many of you, I can easily forget to eat a meal because I'm busy and just don't get hungry as often.