I had this problem as well and some times they would come to the surface and I was able to clear them(yuck!) and other times they would just go away on there own.
Applying a warm compress will help bring the poison to the surface but DO NOT squeeze them unless they are really at the surface because if they break under the skin, they can spread.
Keep the area very clean(an anti bacterial cleanser works fine).
If they persist I would talk to your doctor for advice...sometimes they will lance them for you.
Hope this helps...good luck!