I don't know where the idea that our brain started shrinking before we adopted agriculture came from but if I had to guess, I'd say that it was due to some evidence not coinciding with other evidence and the discrepancy would explain it.

When it comes to anthropological evidence, we must assume that a) we don't know everything that can be known and b) not everything that was known can be known. In other words, even if we find everything that we can find, we still don't have the complete picture of a past when we we're there because some actions didn't leave any evidence for us to find. Having said that, the dominant evidence must be the reference for the weaker one. In further words, whatever we're certain of must be our basis for whatever we're not sure of. We're certain that our brain got bigger because we started to eat meat. But we're not certain that our brain got smaller because of some lack of external stimuli because we don't have the evidence of this lack of stimuli. The stronger evidence must rule the weaker one. The most reasonable explanation for our brain shrinkage is that we started to lack what caused, and maintained, our bigger brain.

Furthermore, we have physiological plausibility to explain the above as well, i.e. we know what is in meat that allows our brain to maintain proper function and size.